2024-03-29T11:22:46.766 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:22:46.767 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --verbose --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721 -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721/orig.config.yaml 2024-03-29T11:22:46.801 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:22:46.880 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622721' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi037.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLoffezH04GjNximqgvmhGPgsmhf2UKf/l7edPWBk4Yd7DA6Ai5XwXMG3gQT7qxKm1AO8CW3cwfU6suMj4+pL/Y= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOgITzAaFXRUJPNt5KLN+jaof4PCZTceu09kE0/gT7yB+Qt2bdVMpg+Ey3GHI4/SO7sl5/Lw3mygW0ryf7Gz3BY= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:22:46.881 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:22:46.881 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:22:46.881 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:22:46.883 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:22:46.906 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:22:46.906 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:22:46.906 INFO:teuthology.packaging:ref: None 2024-03-29T11:22:46.906 INFO:teuthology.packaging:tag: None 2024-03-29T11:22:46.906 INFO:teuthology.packaging:branch: main 2024-03-29T11:22:46.906 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:22:46.906 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:22:47.155 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:22:47.156 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:22:47.163 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:22:47.163 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:22:47.177 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:22:47.190 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:22:47.196 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:22:47.222 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:41.160509', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLoffezH04GjNximqgvmhGPgsmhf2UKf/l7edPWBk4Yd7DA6Ai5XwXMG3gQT7qxKm1AO8CW3cwfU6suMj4+pL/Y='} 2024-03-29T11:22:47.247 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:41.161909', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOgITzAaFXRUJPNt5KLN+jaof4PCZTceu09kE0/gT7yB+Qt2bdVMpg+Ey3GHI4/SO7sl5/Lw3mygW0ryf7Gz3BY='} 2024-03-29T11:22:47.247 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:22:47.253 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:22:47.253 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:22:47.253 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:22:47.326 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f068731f0d0>, signals=[15]) 2024-03-29T11:22:47.327 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:22:47.333 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:22:47.333 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi037.front.sepia.ceph.com 2024-03-29T11:22:47.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:22:47.439 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-03-29T11:22:47.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:22:47.524 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:22:47.531 DEBUG:teuthology.orchestra.run.smithi037:> uname -m 2024-03-29T11:22:47.548 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2024-03-29T11:22:47.548 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2024-03-29T11:22:47.604 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="CentOS Stream" 2024-03-29T11:22:47.604 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="9" 2024-03-29T11:22:47.604 INFO:teuthology.orchestra.run.smithi037.stdout:ID="centos" 2024-03-29T11:22:47.604 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:22:47.604 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="9" 2024-03-29T11:22:47.604 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:22:47.604 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:22:47.604 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31" 2024-03-29T11:22:47.604 INFO:teuthology.orchestra.run.smithi037.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:22:47.604 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:22:47.605 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:22:47.605 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:22:47.605 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:22:47.605 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:22:47.605 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2024-03-29T11:22:47.624 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-03-29T11:22:47.641 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-03-29T11:22:47.641 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-03-29T11:22:47.698 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-03-29T11:22:47.698 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2024-03-29T11:22:47.699 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-03-29T11:22:47.699 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:22:47.699 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2024-03-29T11:22:47.699 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:22:47.699 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:22:47.699 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-03-29T11:22:47.699 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:22:47.699 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:22:47.699 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:22:47.699 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:22:47.699 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:22:47.699 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:22:47.699 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-03-29T11:22:47.723 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:22:47.732 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:22:47.738 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:22:47.738 DEBUG:teuthology.orchestra.run.smithi037:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:22:47.741 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:22:47.758 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:22:47.764 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:22:47.764 DEBUG:teuthology.orchestra.run.smithi037:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:22:47.798 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:22:47.829 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:22:47.884 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:22:47.898 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:22:47.898 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:22:47.898 DEBUG:teuthology.orchestra.run.smithi037:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:22:47.899 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:22:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:22:47.916 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-03-29T11:22:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:22:47.917 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-03-29T11:22:47.970 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:22:47.970 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.14.0-361.el9.x86_64 2024-03-29T11:22:47.970 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum install -y kernel 2024-03-29T11:22:47.974 INFO:teuthology.orchestra.run.smithi037.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:22:47.974 INFO:teuthology.task.kernel:Running kernel on smithi037: 5.14.0-361.el9.x86_64 2024-03-29T11:22:47.974 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum install -y kernel 2024-03-29T11:22:50.798 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - AppStream 6.2 MB/s | 6.7 MB 00:01 2024-03-29T11:22:50.874 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 5.3 MB/s | 6.7 MB 00:01 2024-03-29T11:22:52.999 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - BaseOS 2.0 MB/s | 2.0 MB 00:01 2024-03-29T11:22:53.033 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 1.9 MB/s | 2.0 MB 00:01 2024-03-29T11:22:53.998 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-29T11:22:55.197 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.0 MB 00:01 2024-03-29T11:22:55.624 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 35 MB/s | 19 MB 00:00 2024-03-29T11:22:59.891 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 6.1 MB 00:00 2024-03-29T11:23:01.567 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-03-29T11:23:03.331 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 19 MB 00:07 2024-03-29T11:23:05.435 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 5.5 MB/s | 21 MB 00:03 2024-03-29T11:23:07.341 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-29T11:23:08.800 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-03-29T11:23:10.135 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T11:23:10.502 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T11:23:12.750 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:23:12.751 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:23:12.865 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:23:12.877 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:23:12.877 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-29T11:23:12.877 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:23:12.877 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:23:12.877 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:23:12.877 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-29T11:23:12.877 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:23:12.877 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:23:12.877 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:23:12.878 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:23:12.878 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:23:12.878 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:23:12.878 INFO:teuthology.orchestra.run.smithi146.stdout:Install 4 Packages 2024-03-29T11:23:12.878 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:23:12.878 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 76 M 2024-03-29T11:23:12.879 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 123 M 2024-03-29T11:23:12.879 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:23:13.078 INFO:teuthology.orchestra.run.smithi146.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 963 kB/s | 192 kB 00:00 2024-03-29T11:23:14.562 INFO:teuthology.orchestra.run.smithi146.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 9.0 MB/s | 15 MB 00:01 2024-03-29T11:23:15.237 INFO:teuthology.orchestra.run.smithi146.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 14 MB/s | 34 MB 00:02 2024-03-29T11:23:15.318 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-03-29T11:23:15.653 INFO:teuthology.orchestra.run.smithi146.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 11 MB/s | 28 MB 00:02 2024-03-29T11:23:15.656 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:15.657 INFO:teuthology.orchestra.run.smithi146.stdout:Total 28 MB/s | 76 MB 00:02 2024-03-29T11:23:15.657 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:23:16.010 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:23:16.010 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:23:16.864 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:23:16.865 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:23:17.684 INFO:teuthology.orchestra.run.smithi037.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:23:17.684 INFO:teuthology.orchestra.run.smithi037.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:23:17.784 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:23:17.799 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:23:17.799 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-29T11:23:17.799 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:23:17.799 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:23:17.799 INFO:teuthology.orchestra.run.smithi037.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:23:17.799 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-29T11:23:17.799 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:23:17.799 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:23:17.799 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:23:17.799 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:23:17.799 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:23:17.800 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:23:17.800 INFO:teuthology.orchestra.run.smithi037.stdout:Install 4 Packages 2024-03-29T11:23:17.800 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:23:17.800 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 76 M 2024-03-29T11:23:17.800 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 123 M 2024-03-29T11:23:17.800 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:23:17.962 INFO:teuthology.orchestra.run.smithi037.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:23:18.454 INFO:teuthology.orchestra.run.smithi037.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-29T11:23:19.062 INFO:teuthology.orchestra.run.smithi037.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-03-29T11:23:19.188 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:23:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:23:19.879 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:19.963 INFO:teuthology.orchestra.run.smithi037.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 14 MB/s | 28 MB 00:01 2024-03-29T11:23:19.965 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:19.966 INFO:teuthology.orchestra.run.smithi037.stdout:Total 35 MB/s | 76 MB 00:02 2024-03-29T11:23:19.966 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:23:20.373 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:23:20.374 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:23:21.244 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:23:21.244 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:23:22.673 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:22.813 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:23.662 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:23:24.276 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:23:24.372 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:27.273 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:27.409 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:27.637 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:32.373 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:32.488 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:32.525 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:37.620 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:01.589 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:01.589 INFO:teuthology.orchestra.run.smithi146.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:24:01.589 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:24:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:24:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:24:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:24:02.416 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:24:02.417 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:24:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:24:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:24:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:24:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:24:02.417 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:24:02.786 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:24:02.925 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:02.926 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:24:02.926 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:24:02.926 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:24:02.926 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:24:02.926 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:24:02.926 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:24:02.926 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:24:03.007 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:03.007 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:24:03.007 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2024-03-29T11:24:03.053 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2024-03-29T11:24:03.054 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2024-03-29T11:24:03.054 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.06 2024-03-29T11:24:03.054 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 68.el9 2024-03-29T11:24:03.054 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2024-03-29T11:24:03.054 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:24:03.054 INFO:teuthology.orchestra.run.smithi146.stdout:Group : Unspecified 2024-03-29T11:24:03.054 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 8264254 2024-03-29T11:24:03.054 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2024-03-29T11:24:03.054 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:24:03.054 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:24:03.054 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:24:03.055 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:24:03.055 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : builder@centos.org 2024-03-29T11:24:03.055 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2024-03-29T11:24:03.055 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:24:03.055 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2024-03-29T11:24:03.055 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2024-03-29T11:24:03.055 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:24:03.055 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:24:03.055 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:24:03.055 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:24:03.055 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2024-03-29T11:24:03.055 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:24:03.055 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:24:03.056 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:24:03.056 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:24:03.056 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:24:03.593 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-03-29T11:24:04.970 INFO:teuthology.orchestra.run.smithi146.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:24:05.001 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-03-29T11:24:05.003 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-03-29T11:24:05.018 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.v44DtuW0y2 2024-03-29T11:24:05.019 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.v44DtuW0y2 2024-03-29T11:24:05.086 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.v44DtuW0y2 2024-03-29T11:24:05.228 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.v44DtuW0y2 is 6KB 2024-03-29T11:24:05.277 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.v44DtuW0y2 2024-03-29T11:24:05.291 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:24:05.355 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:24:05.356 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:24:05.356 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:24:05.356 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:24:05.357 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:24:05.522 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-03-29T11:24:08.613 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:08.613 INFO:teuthology.orchestra.run.smithi037.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:24:08.613 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:24:08.657 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:09.014 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:09.015 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:24:09.015 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:24:09.015 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:24:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:24:09.390 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:24:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:24:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:24:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:24:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:24:09.390 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:24:09.728 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:24:09.838 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:09.838 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:24:09.838 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:24:09.838 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:24:09.838 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:24:09.838 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:24:09.838 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi037.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:24:09.838 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:24:09.920 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:09.921 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:24:09.921 DEBUG:teuthology.orchestra.run.smithi037:> sudo rpm -qi grub2-tools 2024-03-29T11:24:09.972 INFO:teuthology.orchestra.run.smithi037.stdout:Name : grub2-tools 2024-03-29T11:24:09.972 INFO:teuthology.orchestra.run.smithi037.stdout:Epoch : 1 2024-03-29T11:24:09.972 INFO:teuthology.orchestra.run.smithi037.stdout:Version : 2.06 2024-03-29T11:24:09.972 INFO:teuthology.orchestra.run.smithi037.stdout:Release : 68.el9 2024-03-29T11:24:09.972 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: x86_64 2024-03-29T11:24:09.972 INFO:teuthology.orchestra.run.smithi037.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:24:09.972 INFO:teuthology.orchestra.run.smithi037.stdout:Group : Unspecified 2024-03-29T11:24:09.972 INFO:teuthology.orchestra.run.smithi037.stdout:Size : 8264254 2024-03-29T11:24:09.972 INFO:teuthology.orchestra.run.smithi037.stdout:License : GPLv3+ 2024-03-29T11:24:09.972 INFO:teuthology.orchestra.run.smithi037.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:24:09.972 INFO:teuthology.orchestra.run.smithi037.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:24:09.973 INFO:teuthology.orchestra.run.smithi037.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:24:09.973 INFO:teuthology.orchestra.run.smithi037.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:24:09.973 INFO:teuthology.orchestra.run.smithi037.stdout:Packager : builder@centos.org 2024-03-29T11:24:09.973 INFO:teuthology.orchestra.run.smithi037.stdout:Vendor : CentOS 2024-03-29T11:24:09.973 INFO:teuthology.orchestra.run.smithi037.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:24:09.973 INFO:teuthology.orchestra.run.smithi037.stdout:Summary : Support tools for GRUB. 2024-03-29T11:24:09.973 INFO:teuthology.orchestra.run.smithi037.stdout:Description : 2024-03-29T11:24:09.973 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:24:09.973 INFO:teuthology.orchestra.run.smithi037.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:24:09.973 INFO:teuthology.orchestra.run.smithi037.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:24:09.973 INFO:teuthology.orchestra.run.smithi037.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:24:09.973 INFO:teuthology.orchestra.run.smithi037.stdout:hardware devices. 2024-03-29T11:24:09.973 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:24:09.973 INFO:teuthology.orchestra.run.smithi037.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:24:09.974 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:24:09.974 INFO:teuthology.task.kernel:Updating grub on smithi037 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:24:09.974 DEBUG:teuthology.orchestra.run.smithi037:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:24:10.542 INFO:teuthology.orchestra.run.smithi037.stderr:Generating grub configuration file ... 2024-03-29T11:24:12.046 INFO:teuthology.orchestra.run.smithi037.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:24:12.087 INFO:teuthology.orchestra.run.smithi037.stderr:done 2024-03-29T11:24:12.088 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-03-29T11:24:12.103 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.XH9qeRnnAy 2024-03-29T11:24:12.104 DEBUG:teuthology.orchestra.run.smithi037:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XH9qeRnnAy 2024-03-29T11:24:12.174 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.XH9qeRnnAy 2024-03-29T11:24:12.313 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.XH9qeRnnAy is 6KB 2024-03-29T11:24:12.362 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.XH9qeRnnAy 2024-03-29T11:24:12.376 DEBUG:teuthology.orchestra.run.smithi037:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:24:12.444 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:24:12.444 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:24:12.444 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:24:12.444 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:24:12.445 DEBUG:teuthology.orchestra.run.smithi037:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:24:12.626 DEBUG:teuthology.orchestra.run.smithi037:> sudo shutdown -r now 2024-03-29T11:24:35.546 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:24:35.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-29T11:24:35.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:42.634 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:24:42.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-29T11:24:42.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:53.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-29T11:25:02.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-29T11:25:02.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:05.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-29T11:25:17.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-29T11:25:17.996 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:42.658 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:25:51.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-29T11:25:51.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:52.021 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-03-29T11:25:52.393 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-29T11:25:52.393 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:25:52.393 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-03-29T11:25:52.409 INFO:teuthology.orchestra.run.smithi037.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:25:52.409 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:25:52.409 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:25:52.409 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:25:53.411 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:25:53.411 DEBUG:teuthology.orchestra.run.smithi037:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:25:53.460 INFO:teuthology.orchestra.run.smithi037.stdout:ttyS1 2024-03-29T11:25:53.499 DEBUG:teuthology.parallel:result is None 2024-03-29T11:26:17.999 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:26:33.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-29T11:26:33.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:26:33.335 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-03-29T11:26:33.721 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-29T11:26:33.721 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:26:33.721 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-03-29T11:26:33.736 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:26:33.736 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:26:33.737 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:26:33.737 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:26:34.738 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:26:34.738 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:26:34.831 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-03-29T11:26:34.871 DEBUG:teuthology.parallel:result is None 2024-03-29T11:26:34.872 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:26:34.883 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:26:34.883 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:26:34.885 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:26:34.902 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:26:34.909 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:26:34.920 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:26:34.920 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:26:34.944 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:26:34.973 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:26:34.981 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:26:34.982 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:26:35.008 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:26:35.041 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:35.046 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:35.056 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:35.060 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:35.062 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:26:35.069 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:26:35.069 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:26:35.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:26:35.133 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:26:35.156 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:26:35.157 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:26:35.171 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:26:35.188 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:35.264 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:35.322 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:35.406 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:35.451 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:26:35.451 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:26:35.520 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:35.564 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:35.624 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:35.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:35.712 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:26:35.713 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:26:35.781 DEBUG:teuthology.orchestra.run.smithi037:> sudo service rsyslog restart 2024-03-29T11:26:35.785 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-03-29T11:26:35.831 INFO:teuthology.orchestra.run.smithi037.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:26:35.868 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:26:36.312 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:26:36.320 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:26:36.320 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:26:36.331 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:26:36.339 DEBUG:teuthology.orchestra.run.smithi037:> sudo service auditd rotate 2024-03-29T11:26:36.399 INFO:teuthology.orchestra.run.smithi037.stdout:Rotating logs: 2024-03-29T11:26:36.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-03-29T11:26:36.463 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-03-29T11:26:36.464 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:26:36.464 DEBUG:teuthology.orchestra.run.smithi037:> /usr/sbin/getenforce 2024-03-29T11:26:36.483 INFO:teuthology.orchestra.run.smithi037.stdout:Permissive 2024-03-29T11:26:36.484 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-03-29T11:26:36.528 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-03-29T11:26:36.529 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi037.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi146.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:26:36.529 DEBUG:teuthology.orchestra.run.smithi037:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:26:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:26:36.559 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:26:36.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:26:36.588 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:26:36.588 DEBUG:teuthology.orchestra.run.smithi037:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:26:36.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:26:36.689 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:26:36.700 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T11:26:36.705 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T11:26:36.829 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:26:36.844 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:26:36.846 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi037.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:31:03.295 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi037.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-03-29T11:31:03.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-29T11:31:03.297 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:31:03.385 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-03-29T11:31:03.473 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-29T11:31:03.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-29T11:31:03.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:31:03.564 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-03-29T11:31:03.644 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-29T11:31:03.645 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:31:03.655 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:31:03.655 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:31:03.656 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:31:03.658 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:31:03.658 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:31:03.688 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:31:03.706 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:31:03.715 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:31:03.732 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:31:03.736 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: ntpd: command not found 2024-03-29T11:31:03.750 INFO:teuthology.orchestra.run.smithi037.stdout:506 Cannot talk to daemon 2024-03-29T11:31:03.762 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-03-29T11:31:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:31:03.776 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-03-29T11:31:03.784 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:31:03.793 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:31:03.809 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:31:03.842 INFO:teuthology.orchestra.run.smithi037.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:31:03.845 INFO:teuthology.orchestra.run.smithi037.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:31:03.845 INFO:teuthology.orchestra.run.smithi037.stdout:=============================================================================== 2024-03-29T11:31:03.845 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:03.846 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:03.846 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:03.846 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:03.861 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:31:03.865 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:31:03.865 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-03-29T11:31:03.865 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:03.865 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:03.865 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:03.865 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:03.866 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:31:03.875 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:31:03.875 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:31:03.875 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:31:03.875 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:31:03.881 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:31:03.881 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:31:03.881 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:31:03.882 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:31:03.884 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:31:03.884 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:31:04.025 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:31:04.025 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:31:04.148 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:31:04.148 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:31:04.153 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:31:04.153 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:31:04.153 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:31:04.181 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:31:04.182 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:31:04.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:31:04.329 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:31:04.395 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2024-03-29T11:31:04.397 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2024-03-29T11:31:04.429 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:31:04.429 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:31:04.429 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:31:04.459 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:31:04.459 DEBUG:teuthology.orchestra.run.smithi037:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:31:04.529 DEBUG:teuthology.orchestra.run.smithi037:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:31:04.609 DEBUG:teuthology.orchestra.run.smithi037:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:31:04.637 INFO:teuthology.orchestra.run.smithi146.stdout:74 files removed 2024-03-29T11:31:04.663 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw 2024-03-29T11:31:04.677 INFO:teuthology.orchestra.run.smithi037.stdout:check_obsoletes = 1 2024-03-29T11:31:04.679 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum clean all 2024-03-29T11:31:04.920 INFO:teuthology.orchestra.run.smithi037.stdout:74 files removed 2024-03-29T11:31:04.947 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-radosgw 2024-03-29T11:31:05.589 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 132 kB/s | 82 kB 00:00 2024-03-29T11:31:05.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph packages for x86_64 147 kB/s | 82 kB 00:00 2024-03-29T11:31:06.115 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-03-29T11:31:06.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph noarch packages 28 kB/s | 10 kB 00:00 2024-03-29T11:31:06.474 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-29T11:31:06.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-29T11:31:06.918 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T11:31:07.107 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T11:31:08.301 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-29T11:31:08.565 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-29T11:31:09.473 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-29T11:31:09.894 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 6.3 MB/s | 8.0 MB 00:01 2024-03-29T11:31:11.309 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-29T11:31:15.358 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-29T11:31:16.884 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-03-29T11:31:18.905 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T11:31:23.998 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-03-29T11:31:26.699 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:31:26.702 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:31:26.702 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-29T11:31:26.702 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:31:26.703 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:31:26.703 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:31:26.703 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrading: 2024-03-29T11:31:26.703 INFO:teuthology.orchestra.run.smithi037.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:31:26.703 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:31:26.703 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-29T11:31:26.703 INFO:teuthology.orchestra.run.smithi037.stdout: boost-program-options 2024-03-29T11:31:26.703 INFO:teuthology.orchestra.run.smithi037.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:31:26.703 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:31:26.703 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:31:26.703 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:31:26.703 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:31:26.703 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:31:26.703 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:31:26.704 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:31:26.704 INFO:teuthology.orchestra.run.smithi037.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:31:26.704 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:31:26.704 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:31:26.704 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:31:26.704 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:31:26.704 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:31:26.704 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:31:26.704 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:31:26.704 INFO:teuthology.orchestra.run.smithi037.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:31:26.704 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse 2024-03-29T11:31:26.704 INFO:teuthology.orchestra.run.smithi037.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:31:26.704 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:31:26.705 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:31:26.705 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:31:26.705 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:31:26.705 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:31:26.705 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:31:26.705 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:31:26.705 INFO:teuthology.orchestra.run.smithi037.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:31:26.705 INFO:teuthology.orchestra.run.smithi037.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:31:26.705 INFO:teuthology.orchestra.run.smithi037.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:31:26.705 INFO:teuthology.orchestra.run.smithi037.stdout:Installing weak dependencies: 2024-03-29T11:31:26.705 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:31:26.705 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:31:26.705 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:31:26.705 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:31:26.706 INFO:teuthology.orchestra.run.smithi037.stdout:Install 30 Packages 2024-03-29T11:31:26.706 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrade 2 Packages 2024-03-29T11:31:26.706 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:31:26.706 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 58 M 2024-03-29T11:31:26.706 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:31:29.995 INFO:teuthology.orchestra.run.smithi037.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.8 MB/s | 5.3 MB 00:03 2024-03-29T11:31:30.087 INFO:teuthology.orchestra.run.smithi037.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 275 kB/s | 25 kB 00:00 2024-03-29T11:31:30.496 INFO:teuthology.orchestra.run.smithi037.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.8 MB/s | 742 kB 00:00 2024-03-29T11:31:30.779 INFO:teuthology.orchestra.run.smithi037.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.7 MB/s | 498 kB 00:00 2024-03-29T11:31:32.668 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 891 kB/s | 19 MB 00:21 2024-03-29T11:31:33.446 INFO:teuthology.orchestra.run.smithi037.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.5 MB/s | 9.7 MB 00:06 2024-03-29T11:31:33.530 INFO:teuthology.orchestra.run.smithi037.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 543 kB/s | 45 kB 00:00 2024-03-29T11:31:33.597 INFO:teuthology.orchestra.run.smithi037.stdout:(7/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 1.8 MB/s | 5.1 MB 00:02 2024-03-29T11:31:33.722 INFO:teuthology.orchestra.run.smithi037.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 685 kB/s | 131 kB 00:00 2024-03-29T11:31:33.947 INFO:teuthology.orchestra.run.smithi037.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 1.4 MB/s | 320 kB 00:00 2024-03-29T11:31:33.972 INFO:teuthology.orchestra.run.smithi037.stdout:(10/32): python3-cephfs-19.0.0-1439.g5dcb190d.e 431 kB/s | 162 kB 00:00 2024-03-29T11:31:34.089 INFO:teuthology.orchestra.run.smithi037.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 851 kB/s | 99 kB 00:00 2024-03-29T11:31:34.181 INFO:teuthology.orchestra.run.smithi037.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 298 kB 00:00 2024-03-29T11:31:34.206 INFO:teuthology.orchestra.run.smithi037.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 906 kB/s | 106 kB 00:00 2024-03-29T11:31:34.232 INFO:teuthology.orchestra.run.smithi037.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-29T11:31:34.257 INFO:teuthology.orchestra.run.smithi037.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-29T11:31:34.291 INFO:teuthology.orchestra.run.smithi037.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-29T11:31:34.316 INFO:teuthology.orchestra.run.smithi037.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-29T11:31:34.341 INFO:teuthology.orchestra.run.smithi037.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2024-03-29T11:31:34.367 INFO:teuthology.orchestra.run.smithi037.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:31:34.392 INFO:teuthology.orchestra.run.smithi037.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 692 kB/s | 35 kB 00:00 2024-03-29T11:31:34.409 INFO:teuthology.orchestra.run.smithi037.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-29T11:31:34.934 INFO:teuthology.orchestra.run.smithi037.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 570 kB/s | 309 kB 00:00 2024-03-29T11:31:35.026 INFO:teuthology.orchestra.run.smithi037.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2024-03-29T11:31:35.127 INFO:teuthology.orchestra.run.smithi037.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-03-29T11:31:35.236 INFO:teuthology.orchestra.run.smithi037.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.3 MB/s | 4.4 MB 00:00 2024-03-29T11:31:35.261 INFO:teuthology.orchestra.run.smithi037.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-03-29T11:31:35.344 INFO:teuthology.orchestra.run.smithi037.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 838 kB 00:00 2024-03-29T11:31:35.382 INFO:teuthology.orchestra.run.smithi037.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-29T11:31:35.462 INFO:teuthology.orchestra.run.smithi037.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-29T11:31:37.487 INFO:teuthology.orchestra.run.smithi037.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 3.1 MB 00:02 2024-03-29T11:31:38.054 INFO:teuthology.orchestra.run.smithi037.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 3.4 MB 00:02 2024-03-29T11:31:38.852 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.1 MB 00:02 2024-03-29T11:31:40.662 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-29T11:31:43.355 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 21 MB 00:02 2024-03-29T11:31:48.238 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-29T11:31:50.859 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:31:50.870 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:31:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-29T11:31:50.870 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:31:50.870 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:31:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:31:50.870 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2024-03-29T11:31:50.870 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:31:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:31:50.871 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-29T11:31:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options 2024-03-29T11:31:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:31:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:31:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:31:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:31:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:31:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:31:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:31:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:31:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:31:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:31:50.871 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:31:50.872 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:31:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:31:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:31:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:31:50.873 INFO:teuthology.orchestra.run.smithi146.stdout:Installing weak dependencies: 2024-03-29T11:31:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:31:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:31:50.873 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:31:50.873 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:31:50.873 INFO:teuthology.orchestra.run.smithi146.stdout:Install 30 Packages 2024-03-29T11:31:50.873 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 2 Packages 2024-03-29T11:31:50.873 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:31:50.874 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 58 M 2024-03-29T11:31:50.874 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:31:51.537 INFO:teuthology.orchestra.run.smithi037.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 828 kB/s | 20 MB 00:24 2024-03-29T11:31:51.541 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:51.541 INFO:teuthology.orchestra.run.smithi037.stdout:Total 2.3 MB/s | 58 MB 00:24 2024-03-29T11:31:51.541 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:31:51.585 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:31:51.585 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:31:52.076 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:31:52.076 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:31:53.730 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:31:53.822 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:31:53.858 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:31:53.883 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:31:53.906 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:53.923 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:54.073 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:54.129 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:54.470 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:31:54.518 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:54.593 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:54.684 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:31:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:54.891 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:54.919 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:31:54.964 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:55.055 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:55.119 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:55.154 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi146.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 2.3 MB/s | 9.7 MB 00:04 2024-03-29T11:31:55.264 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:31:55.322 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:31:55.322 INFO:teuthology.orchestra.run.smithi146.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 303 kB/s | 25 kB 00:00 2024-03-29T11:31:55.370 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:31:55.403 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:31:55.464 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:31:55.631 INFO:teuthology.orchestra.run.smithi146.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 2.4 MB/s | 742 kB 00:00 2024-03-29T11:31:55.792 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:31:55.819 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:55.831 INFO:teuthology.orchestra.run.smithi146.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.4 MB/s | 498 kB 00:00 2024-03-29T11:31:55.892 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:55.924 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:31:55.967 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:31:56.448 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:31:56.472 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:56.552 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:56.621 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:31:56.649 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:31:57.381 INFO:teuthology.orchestra.run.smithi146.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 3.3 MB/s | 5.1 MB 00:01 2024-03-29T11:31:57.456 INFO:teuthology.orchestra.run.smithi146.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 604 kB/s | 45 kB 00:00 2024-03-29T11:31:57.548 INFO:teuthology.orchestra.run.smithi146.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.4 MB/s | 131 kB 00:00 2024-03-29T11:31:57.632 INFO:teuthology.orchestra.run.smithi146.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.9 MB/s | 162 kB 00:00 2024-03-29T11:31:57.775 INFO:teuthology.orchestra.run.smithi146.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 2.2 MB/s | 320 kB 00:00 2024-03-29T11:31:57.916 INFO:teuthology.orchestra.run.smithi146.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.1 MB/s | 298 kB 00:00 2024-03-29T11:31:58.008 INFO:teuthology.orchestra.run.smithi146.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 99 kB 00:00 2024-03-29T11:31:58.193 INFO:teuthology.orchestra.run.smithi146.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 578 kB/s | 106 kB 00:00 2024-03-29T11:31:58.284 INFO:teuthology.orchestra.run.smithi146.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 159 kB 00:00 2024-03-29T11:31:58.317 INFO:teuthology.orchestra.run.smithi146.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T11:31:58.393 INFO:teuthology.orchestra.run.smithi146.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-29T11:31:58.434 INFO:teuthology.orchestra.run.smithi146.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-29T11:31:58.459 INFO:teuthology.orchestra.run.smithi146.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-29T11:31:58.494 INFO:teuthology.orchestra.run.smithi146.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 6.5 MB/s | 220 kB 00:00 2024-03-29T11:31:58.519 INFO:teuthology.orchestra.run.smithi146.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.4 MB/s | 35 kB 00:00 2024-03-29T11:31:58.543 INFO:teuthology.orchestra.run.smithi146.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-03-29T11:31:58.719 INFO:teuthology.orchestra.run.smithi146.stdout:(21/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x8 715 kB/s | 5.3 MB 00:07 2024-03-29T11:31:58.881 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:58.922 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:58.969 INFO:teuthology.orchestra.run.smithi146.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 726 kB/s | 309 kB 00:00 2024-03-29T11:31:59.128 INFO:teuthology.orchestra.run.smithi146.stdout:(23/32): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-29T11:31:59.153 INFO:teuthology.orchestra.run.smithi146.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 142 kB/s | 26 kB 00:00 2024-03-29T11:31:59.187 INFO:teuthology.orchestra.run.smithi146.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 832 kB/s | 49 kB 00:00 2024-03-29T11:31:59.221 INFO:teuthology.orchestra.run.smithi146.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-29T11:31:59.263 INFO:teuthology.orchestra.run.smithi146.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-03-29T11:31:59.288 INFO:teuthology.orchestra.run.smithi146.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-29T11:31:59.354 INFO:teuthology.orchestra.run.smithi146.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-03-29T11:31:59.503 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:59.541 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:59.644 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:59.644 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:31:59.645 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:31:59.645 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:31:59.667 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:00.754 INFO:teuthology.orchestra.run.smithi146.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 3.1 MB 00:01 2024-03-29T11:32:04.647 INFO:teuthology.orchestra.run.smithi146.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 645 kB/s | 3.4 MB 00:05 2024-03-29T11:32:05.331 INFO:teuthology.orchestra.run.smithi146.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.4 MB/s | 20 MB 00:14 2024-03-29T11:32:05.335 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:05.335 INFO:teuthology.orchestra.run.smithi146.stdout:Total 4.0 MB/s | 58 MB 00:14 2024-03-29T11:32:05.335 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:32:05.373 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:32:05.373 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:32:05.849 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:32:05.850 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:32:07.468 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:32:07.557 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:32:07.584 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:32:07.610 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:32:07.632 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:32:07.648 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:32:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:32:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:32:08.271 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:32:08.311 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:32:08.401 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:32:08.490 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:32:08.512 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:32:08.584 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:32:08.612 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:32:08.656 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:32:08.724 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:32:08.748 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:32:08.795 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:32:08.832 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:32:08.937 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:32:08.974 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:32:09.025 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:32:09.055 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:32:09.082 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:32:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:32:09.284 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:09.284 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /sys 2024-03-29T11:32:09.284 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /proc 2024-03-29T11:32:09.285 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /mnt 2024-03-29T11:32:09.285 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /var/tmp 2024-03-29T11:32:09.285 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /home 2024-03-29T11:32:09.285 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /root 2024-03-29T11:32:09.285 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /tmp 2024-03-29T11:32:09.285 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:32:09.424 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:32:09.445 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:32:09.534 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:32:09.567 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:32:09.607 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:32:10.060 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:32:10.089 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:32:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:32:10.202 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:32:10.211 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:32:10.226 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:32:10.226 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:10.226 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:32:10.226 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:32:10.226 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:32:10.226 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:32:10.228 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:32:10.241 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:32:10.296 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:32:10.296 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:11.397 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:11.397 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:32:11.397 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:32:11.397 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:32:11.397 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:32:11.397 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:32:11.397 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:32:11.397 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:32:11.397 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:32:11.397 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:32:11.398 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:32:11.398 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:32:11.398 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:32:11.398 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:32:11.398 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:32:11.398 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:32:11.398 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:32:11.398 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:32:11.398 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:32:11.398 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:32:11.398 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:32:11.398 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:32:11.398 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:32:11.398 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:32:11.399 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:32:11.399 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:32:11.399 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:32:11.399 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:32:11.399 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:32:11.399 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:32:11.399 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:32:11.399 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:11.399 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:32:11.399 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:32:12.380 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:12.381 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:32:12.381 INFO:teuthology.orchestra.run.smithi037.stdout:Upgraded: 2024-03-29T11:32:12.381 INFO:teuthology.orchestra.run.smithi037.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.381 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.381 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:32:12.381 INFO:teuthology.orchestra.run.smithi037.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:32:12.381 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.381 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.381 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.381 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.381 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:32:12.381 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:32:12.382 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:32:12.382 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.382 INFO:teuthology.orchestra.run.smithi037.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:32:12.382 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:32:12.382 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:32:12.382 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.382 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:32:12.382 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.382 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:32:12.382 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:32:12.382 INFO:teuthology.orchestra.run.smithi037.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:32:12.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:32:12.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.383 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:32:12.383 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:32:12.383 INFO:teuthology.orchestra.run.smithi037.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:32:12.383 INFO:teuthology.orchestra.run.smithi037.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:32:12.383 INFO:teuthology.orchestra.run.smithi037.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:32:12.383 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:32:12.383 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:32:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:32:12.421 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:32:12.690 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-test 2024-03-29T11:32:12.980 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:32:13.015 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:32:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:32:13.091 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:32:13.091 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:32:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:32:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:13.261 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:50 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:32:13.422 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:32:13.423 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:32:13.423 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-29T11:32:13.423 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:32:13.423 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:32:13.423 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:32:13.423 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-29T11:32:13.423 INFO:teuthology.orchestra.run.smithi037.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:32:13.423 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:32:13.424 INFO:teuthology.orchestra.run.smithi037.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:32:13.424 INFO:teuthology.orchestra.run.smithi037.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:32:13.424 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:32:13.424 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:32:13.424 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:32:13.424 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:32:13.424 INFO:teuthology.orchestra.run.smithi037.stdout:Install 6 Packages 2024-03-29T11:32:13.424 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:32:13.425 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 48 M 2024-03-29T11:32:13.425 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 204 M 2024-03-29T11:32:13.425 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:32:13.700 INFO:teuthology.orchestra.run.smithi037.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-29T11:32:13.766 INFO:teuthology.orchestra.run.smithi037.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-29T11:32:13.792 INFO:teuthology.orchestra.run.smithi037.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-29T11:32:14.059 INFO:teuthology.orchestra.run.smithi037.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 328 kB/s | 163 kB 00:00 2024-03-29T11:32:14.301 INFO:teuthology.orchestra.run.smithi037.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 369 kB/s | 187 kB 00:00 2024-03-29T11:32:22.482 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:22.482 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2024-03-29T11:32:22.482 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2024-03-29T11:32:22.482 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2024-03-29T11:32:22.482 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2024-03-29T11:32:22.482 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2024-03-29T11:32:22.482 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /root 2024-03-29T11:32:22.482 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2024-03-29T11:32:22.482 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:32:23.356 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:32:23.381 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:32:23.381 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:23.381 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:32:23.381 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:32:23.381 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:32:23.381 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:32:23.382 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:32:23.458 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:32:23.459 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:32:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:32:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:32:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:32:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:32:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:32:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:32:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:32:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:32:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:32:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:32:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:32:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:32:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:32:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:32:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:32:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:32:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:32:25.857 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:32:25.858 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2024-03-29T11:32:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.858 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:32:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:32:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:32:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:32:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:32:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:32:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:32:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:32:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:32:25.860 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:32:26.204 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-test 2024-03-29T11:32:26.766 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:38 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:32:26.918 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:32:26.919 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:32:26.919 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-29T11:32:26.919 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:32:26.919 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:32:26.919 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:32:26.919 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-29T11:32:26.919 INFO:teuthology.orchestra.run.smithi146.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:32:26.919 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:32:26.919 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:32:26.919 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:32:26.919 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:32:26.920 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:32:26.920 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:32:26.920 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:32:26.920 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2024-03-29T11:32:26.920 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:32:26.920 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 48 M 2024-03-29T11:32:26.920 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 204 M 2024-03-29T11:32:26.921 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:32:27.086 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-03-29T11:32:27.153 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.7 MB/s | 300 kB 00:00 2024-03-29T11:32:27.195 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-29T11:32:27.336 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-29T11:32:27.870 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 179 kB/s | 163 kB 00:00 2024-03-29T11:32:46.632 INFO:teuthology.orchestra.run.smithi037.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 47 MB 00:33 2024-03-29T11:32:46.634 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:46.635 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.5 MB/s | 48 MB 00:33 2024-03-29T11:32:46.648 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:32:46.671 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:32:46.671 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:32:47.071 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:32:47.072 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:32:47.352 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 2.3 MB/s | 47 MB 00:20 2024-03-29T11:32:47.355 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:47.355 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.4 MB/s | 48 MB 00:20 2024-03-29T11:32:47.370 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:32:47.392 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:32:47.392 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:32:47.675 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:32:47.741 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:32:47.780 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:32:47.780 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:32:47.783 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:32:47.823 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:47.883 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:32:47.912 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:32:48.436 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:32:48.485 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:32:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:32:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:52.654 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:52.731 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:53.089 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:53.171 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:55.152 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:55.153 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:32:55.153 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:32:55.153 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:55.153 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:32:55.153 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:32:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:32:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:32:55.531 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:32:55.531 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:32:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:32:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:32:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:32:55.532 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:32:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:32:55.536 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:55.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:32:55.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:32:55.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:55.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:32:55.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:32:55.800 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph 2024-03-29T11:32:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:32:55.923 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:32:55.924 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:32:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:32:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:32:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:32:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:32:55.924 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:32:55.924 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:32:56.245 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph 2024-03-29T11:32:56.366 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:32:56.527 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:32:56.530 INFO:teuthology.orchestra.run.smithi037.stdout:========================================================================================= 2024-03-29T11:32:56.530 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-29T11:32:56.530 INFO:teuthology.orchestra.run.smithi037.stdout:========================================================================================= 2024-03-29T11:32:56.530 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:32:56.530 INFO:teuthology.orchestra.run.smithi037.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:32:56.530 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-29T11:32:56.530 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:32:56.530 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:32:56.530 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:32:56.530 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:32:56.530 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:32:56.530 INFO:teuthology.orchestra.run.smithi037.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:32:56.531 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:32:56.532 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:32:56.533 INFO:teuthology.orchestra.run.smithi037.stdout:========================================================================================= 2024-03-29T11:32:56.533 INFO:teuthology.orchestra.run.smithi037.stdout:Install 36 Packages 2024-03-29T11:32:56.533 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:32:56.533 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 30 M 2024-03-29T11:32:56.533 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 107 M 2024-03-29T11:32:56.533 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:32:56.820 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:32:56.978 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:32:56.980 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2024-03-29T11:32:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-29T11:32:56.980 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:32:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:32:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:32:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:32:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:32:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:32:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:32:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:32:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:32:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:32:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:32:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:32:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:32:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi146.stdout:Install 36 Packages 2024-03-29T11:32:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:32:56.984 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 30 M 2024-03-29T11:32:56.984 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 107 M 2024-03-29T11:32:56.984 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:32:57.064 INFO:teuthology.orchestra.run.smithi037.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 24 kB/s | 6.6 kB 00:00 2024-03-29T11:32:57.615 INFO:teuthology.orchestra.run.smithi146.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 23 kB/s | 6.6 kB 00:00 2024-03-29T11:32:58.280 INFO:teuthology.orchestra.run.smithi037.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 2.1 MB 00:01 2024-03-29T11:32:58.764 INFO:teuthology.orchestra.run.smithi037.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 770 kB/s | 1.5 MB 00:01 2024-03-29T11:32:58.832 INFO:teuthology.orchestra.run.smithi146.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 2.1 MB 00:01 2024-03-29T11:32:59.072 INFO:teuthology.orchestra.run.smithi037.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 803 kB/s | 247 kB 00:00 2024-03-29T11:32:59.264 INFO:teuthology.orchestra.run.smithi037.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 974 kB/s | 187 kB 00:00 2024-03-29T11:32:59.340 INFO:teuthology.orchestra.run.smithi037.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-29T11:32:59.382 INFO:teuthology.orchestra.run.smithi037.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 758 kB/s | 32 kB 00:00 2024-03-29T11:32:59.440 INFO:teuthology.orchestra.run.smithi037.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 810 kB/s | 47 kB 00:00 2024-03-29T11:32:59.474 INFO:teuthology.orchestra.run.smithi037.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:32:59.533 INFO:teuthology.orchestra.run.smithi037.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-29T11:32:59.649 INFO:teuthology.orchestra.run.smithi037.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 9.8 MB/s | 1.1 MB 00:00 2024-03-29T11:32:59.716 INFO:teuthology.orchestra.run.smithi037.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-29T11:32:59.775 INFO:teuthology.orchestra.run.smithi037.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-03-29T11:32:59.817 INFO:teuthology.orchestra.run.smithi037.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-29T11:33:00.609 INFO:teuthology.orchestra.run.smithi037.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 158 kB/s | 125 kB 00:00 2024-03-29T11:33:00.876 INFO:teuthology.orchestra.run.smithi037.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 84 kB/s | 22 kB 00:00 2024-03-29T11:33:00.990 INFO:teuthology.orchestra.run.smithi146.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 415 kB/s | 1.5 MB 00:03 2024-03-29T11:33:01.449 INFO:teuthology.orchestra.run.smithi146.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 540 kB/s | 247 kB 00:00 2024-03-29T11:33:01.565 INFO:teuthology.orchestra.run.smithi146.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:33:01.608 INFO:teuthology.orchestra.run.smithi146.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:33:01.641 INFO:teuthology.orchestra.run.smithi146.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 954 kB/s | 32 kB 00:00 2024-03-29T11:33:01.675 INFO:teuthology.orchestra.run.smithi146.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:33:01.708 INFO:teuthology.orchestra.run.smithi146.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:33:01.750 INFO:teuthology.orchestra.run.smithi146.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:33:01.817 INFO:teuthology.orchestra.run.smithi146.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:33:01.851 INFO:teuthology.orchestra.run.smithi146.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:33:01.851 INFO:teuthology.orchestra.run.smithi037.stdout:(17/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 4.7 MB 00:04 2024-03-29T11:33:01.884 INFO:teuthology.orchestra.run.smithi146.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:33:01.918 INFO:teuthology.orchestra.run.smithi146.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:33:01.993 INFO:teuthology.orchestra.run.smithi146.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.6 MB/s | 125 kB 00:00 2024-03-29T11:33:02.085 INFO:teuthology.orchestra.run.smithi037.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 125 kB/s | 151 kB 00:01 2024-03-29T11:33:02.177 INFO:teuthology.orchestra.run.smithi037.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-03-29T11:33:02.302 INFO:teuthology.orchestra.run.smithi037.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-03-29T11:33:02.385 INFO:teuthology.orchestra.run.smithi037.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-29T11:33:02.444 INFO:teuthology.orchestra.run.smithi037.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-03-29T11:33:02.503 INFO:teuthology.orchestra.run.smithi037.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 397 kB/s | 23 kB 00:00 2024-03-29T11:33:02.568 INFO:teuthology.orchestra.run.smithi146.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-03-29T11:33:02.578 INFO:teuthology.orchestra.run.smithi037.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2024-03-29T11:33:02.637 INFO:teuthology.orchestra.run.smithi037.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 337 kB/s | 20 kB 00:00 2024-03-29T11:33:02.703 INFO:teuthology.orchestra.run.smithi037.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 695 kB/s | 46 kB 00:00 2024-03-29T11:33:02.779 INFO:teuthology.orchestra.run.smithi037.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-29T11:33:02.835 INFO:teuthology.orchestra.run.smithi146.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 567 kB/s | 151 kB 00:00 2024-03-29T11:33:02.879 INFO:teuthology.orchestra.run.smithi037.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-29T11:33:02.894 INFO:teuthology.orchestra.run.smithi146.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 747 kB/s | 43 kB 00:00 2024-03-29T11:33:02.929 INFO:teuthology.orchestra.run.smithi037.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 329 kB/s | 16 kB 00:00 2024-03-29T11:33:02.996 INFO:teuthology.orchestra.run.smithi037.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-29T11:33:03.002 INFO:teuthology.orchestra.run.smithi146.stdout:(19/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 890 kB/s | 4.7 MB 00:05 2024-03-29T11:33:03.013 INFO:teuthology.orchestra.run.smithi037.stdout:(31/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 37 kB/s | 43 kB 00:01 2024-03-29T11:33:03.022 INFO:teuthology.orchestra.run.smithi146.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-03-29T11:33:03.055 INFO:teuthology.orchestra.run.smithi037.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 612 kB/s | 36 kB 00:00 2024-03-29T11:33:03.088 INFO:teuthology.orchestra.run.smithi146.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 161 kB/s | 11 kB 00:00 2024-03-29T11:33:03.139 INFO:teuthology.orchestra.run.smithi037.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-29T11:33:03.164 INFO:teuthology.orchestra.run.smithi146.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-29T11:33:03.172 INFO:teuthology.orchestra.run.smithi037.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-03-29T11:33:03.189 INFO:teuthology.orchestra.run.smithi037.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-03-29T11:33:03.231 INFO:teuthology.orchestra.run.smithi146.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 347 kB/s | 23 kB 00:00 2024-03-29T11:33:03.306 INFO:teuthology.orchestra.run.smithi146.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-03-29T11:33:03.331 INFO:teuthology.orchestra.run.smithi146.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-29T11:33:03.373 INFO:teuthology.orchestra.run.smithi146.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2024-03-29T11:33:03.399 INFO:teuthology.orchestra.run.smithi146.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 693 kB/s | 46 kB 00:00 2024-03-29T11:33:03.449 INFO:teuthology.orchestra.run.smithi146.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-29T11:33:03.482 INFO:teuthology.orchestra.run.smithi146.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-29T11:33:03.516 INFO:teuthology.orchestra.run.smithi146.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2024-03-29T11:33:03.550 INFO:teuthology.orchestra.run.smithi146.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-29T11:33:03.583 INFO:teuthology.orchestra.run.smithi146.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 534 kB/s | 36 kB 00:00 2024-03-29T11:33:03.625 INFO:teuthology.orchestra.run.smithi146.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-29T11:33:03.692 INFO:teuthology.orchestra.run.smithi146.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 300 kB/s | 20 kB 00:00 2024-03-29T11:33:03.717 INFO:teuthology.orchestra.run.smithi146.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-29T11:33:06.959 INFO:teuthology.orchestra.run.smithi146.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 2.1 MB/s | 17 MB 00:08 2024-03-29T11:33:06.963 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:06.963 INFO:teuthology.orchestra.run.smithi146.stdout:Total 3.0 MB/s | 30 MB 00:09 2024-03-29T11:33:06.989 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:33:07.032 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:33:07.033 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:33:07.443 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:33:07.443 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:33:08.778 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:33:08.845 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:33:08.904 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:33:08.939 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:33:09.009 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:33:09.062 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:33:09.096 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:33:09.188 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:33:09.274 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:33:09.315 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:33:09.387 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:33:09.440 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:33:09.479 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:33:09.536 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:33:09.591 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:33:09.776 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:33:09.824 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:33:09.863 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:33:09.898 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:33:09.928 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:33:10.005 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:33:10.038 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:33:10.071 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:33:10.112 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:33:10.146 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:33:10.206 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:33:10.296 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:33:10.337 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:33:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:33:10.539 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:33:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:33:10.830 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:33:10.858 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:33:10.884 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:33:10.884 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:10.885 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:33:10.885 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:33:10.885 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:33:10.885 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:11.206 INFO:teuthology.orchestra.run.smithi037.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.3 MB/s | 17 MB 00:12 2024-03-29T11:33:11.209 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:11.209 INFO:teuthology.orchestra.run.smithi037.stdout:Total 2.0 MB/s | 30 MB 00:14 2024-03-29T11:33:11.233 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:33:11.281 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:33:11.281 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:33:11.723 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:33:11.723 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:33:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:33:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:33:12.344 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:12.345 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:33:12.345 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:33:12.345 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:33:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:12.784 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:33:12.808 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:33:12.808 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:12.808 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:33:12.808 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:33:12.808 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:33:12.809 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:13.027 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:33:13.051 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:33:13.051 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:13.051 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:33:13.051 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:33:13.051 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:33:13.051 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:13.091 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:33:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:33:13.141 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:33:13.183 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:33:13.210 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:33:13.284 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:33:13.326 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:33:13.359 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:33:13.455 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:33:13.532 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:33:13.588 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:33:13.661 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:33:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:33:13.750 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:33:13.809 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:33:13.881 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:33:14.075 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:33:14.121 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:33:14.160 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:33:14.194 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:33:14.249 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:33:14.320 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:33:14.351 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:33:14.384 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:33:14.425 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:33:14.469 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:33:14.511 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:33:14.605 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:33:14.642 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:33:14.718 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:33:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:33:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:33:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:33:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:33:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:33:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:33:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:33:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:33:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:33:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:33:14.799 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:33:14.800 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:33:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:33:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:33:14.801 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:33:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:33:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:33:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:33:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:33:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:33:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:33:14.802 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:33:14.859 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:33:14.992 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:33:15.158 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:33:15.188 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:33:15.213 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:33:15.213 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:15.213 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:33:15.213 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:33:15.213 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:33:15.213 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:15.865 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:33:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:15.866 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:33:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:33:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:33:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:33:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:33:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:33:15.868 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:33:15.868 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:33:15.868 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:33:15.868 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:33:15.868 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:33:15.868 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:33:15.868 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:33:15.868 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:33:15.868 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:33:15.868 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:33:15.868 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:15.868 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:33:16.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-base 2024-03-29T11:33:16.731 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:33:16.756 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:33:16.756 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:16.756 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:33:16.756 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:33:16.756 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:33:16.756 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:16.843 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:33:16.941 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:16.993 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:33:16.994 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-29T11:33:16.994 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:33:17.051 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install cephadm 2024-03-29T11:33:17.215 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:33:17.237 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:33:17.237 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:17.237 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:33:17.237 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:33:17.237 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:33:17.238 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:17.457 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:33:17.479 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:33:17.480 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:17.480 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:33:17.480 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:33:17.480 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:33:17.480 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:17.548 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:33:17.613 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:33:17.763 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:33:17.764 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:17.764 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-29T11:33:17.764 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:17.764 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:33:17.765 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:33:17.765 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:17.765 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:33:17.765 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:17.765 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-29T11:33:17.765 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:17.765 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 747 k 2024-03-29T11:33:17.765 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 754 k 2024-03-29T11:33:17.765 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:33:18.198 INFO:teuthology.orchestra.run.smithi146.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.7 MB/s | 747 kB 00:00 2024-03-29T11:33:18.199 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:18.199 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.7 MB/s | 747 kB 00:00 2024-03-29T11:33:18.199 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:33:18.204 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:33:18.205 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:33:18.210 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:33:18.210 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:33:18.282 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:33:18.724 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:18.812 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:19.231 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:33:19.231 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:33:19.231 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:33:19.231 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:33:19.231 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:33:19.232 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:33:19.233 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:33:19.233 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:33:19.233 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:33:19.233 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:33:19.233 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:33:19.233 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:33:19.233 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:33:19.233 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:33:19.233 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:33:19.233 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:33:19.233 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:33:19.233 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:33:19.233 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:33:19.233 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:33:19.251 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:19.538 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:33:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:19.538 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:33:19.745 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:33:20.147 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:33:20.148 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:20.148 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:33:20.148 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:20.148 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:20.148 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:20.148 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:20.148 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:20.148 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:20.148 INFO:teuthology.orchestra.run.smithi037.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:33:20.148 INFO:teuthology.orchestra.run.smithi037.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:33:20.148 INFO:teuthology.orchestra.run.smithi037.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:33:20.148 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:33:20.149 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:33:20.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:33:20.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:33:20.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:33:20.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:33:20.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:33:20.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:33:20.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:33:20.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:33:20.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:33:20.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:33:20.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:33:20.150 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:20.150 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:33:20.321 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:33:20.476 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:33:20.477 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-29T11:33:20.477 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:20.477 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:33:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:33:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:20.477 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:33:20.477 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:20.477 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-29T11:33:20.477 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:20.478 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 145 k 2024-03-29T11:33:20.478 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 436 k 2024-03-29T11:33:20.478 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:33:20.493 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-base 2024-03-29T11:33:20.696 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 668 kB/s | 145 kB 00:00 2024-03-29T11:33:20.696 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:20.696 INFO:teuthology.orchestra.run.smithi146.stdout:Total 664 kB/s | 145 kB 00:00 2024-03-29T11:33:20.697 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:33:20.704 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:33:20.704 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:33:20.758 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:33:20.758 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:33:20.905 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:33:20.963 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:20.987 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:20.987 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:20.987 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:33:20.987 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:21.079 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:33:21.182 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:21.238 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:33:21.239 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-29T11:33:21.240 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:33:21.314 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install cephadm 2024-03-29T11:33:21.883 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:33:21.921 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:21.921 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:21.921 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:33:21.921 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:21.921 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:21.921 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:33:22.039 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:33:22.040 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:33:22.040 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-29T11:33:22.040 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:33:22.040 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:33:22.040 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:33:22.040 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:22.041 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:33:22.041 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:33:22.041 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-29T11:33:22.041 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:22.041 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 747 k 2024-03-29T11:33:22.041 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 754 k 2024-03-29T11:33:22.041 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:33:22.172 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr 2024-03-29T11:33:22.337 INFO:teuthology.orchestra.run.smithi037.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.5 MB/s | 747 kB 00:00 2024-03-29T11:33:22.338 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:22.338 INFO:teuthology.orchestra.run.smithi037.stdout:Total 2.5 MB/s | 747 kB 00:00 2024-03-29T11:33:22.338 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:33:22.344 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:33:22.344 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:33:22.350 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:33:22.350 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:33:22.429 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:33:22.752 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:33:22.814 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:22.894 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:33:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-29T11:33:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:33:22.968 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:33:23.262 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:23.522 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:23.522 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:23.522 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:33:23.522 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:23.522 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:23.523 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:33:23.538 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:33:23.693 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:33:23.694 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-03-29T11:33:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:33:23.695 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:33:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:23.696 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:33:23.696 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-03-29T11:33:23.696 INFO:teuthology.orchestra.run.smithi146.stdout:Install 12 Packages 2024-03-29T11:33:23.696 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:23.696 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 6.9 M 2024-03-29T11:33:23.696 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 26 M 2024-03-29T11:33:23.697 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:33:23.736 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:33:24.242 INFO:teuthology.orchestra.run.smithi146.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 79 kB/s | 15 kB 00:00 2024-03-29T11:33:24.300 INFO:teuthology.orchestra.run.smithi146.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 124 kB/s | 31 kB 00:00 2024-03-29T11:33:24.311 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:33:24.384 INFO:teuthology.orchestra.run.smithi146.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-29T11:33:24.417 INFO:teuthology.orchestra.run.smithi146.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-29T11:33:24.468 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:33:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:33:24.469 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-29T11:33:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:33:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:33:24.469 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:33:24.469 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:33:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:33:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-29T11:33:24.469 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:24.470 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 145 k 2024-03-29T11:33:24.470 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 436 k 2024-03-29T11:33:24.470 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:33:24.618 INFO:teuthology.orchestra.run.smithi146.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.6 MB/s | 863 kB 00:00 2024-03-29T11:33:24.693 INFO:teuthology.orchestra.run.smithi146.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.8 MB/s | 1.8 MB 00:00 2024-03-29T11:33:24.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 407 kB/s | 145 kB 00:00 2024-03-29T11:33:24.827 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:24.828 INFO:teuthology.orchestra.run.smithi037.stdout:Total 405 kB/s | 145 kB 00:00 2024-03-29T11:33:24.828 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:33:24.836 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:33:24.836 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:33:24.891 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:33:24.891 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:33:24.985 INFO:teuthology.orchestra.run.smithi146.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 53 kB/s | 19 kB 00:00 2024-03-29T11:33:25.036 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:33:25.094 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:25.119 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:25.119 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:25.119 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:33:25.119 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:25.144 INFO:teuthology.orchestra.run.smithi146.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 755 kB/s | 548 kB 00:00 2024-03-29T11:33:25.269 INFO:teuthology.orchestra.run.smithi146.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 246 kB/s | 31 kB 00:00 2024-03-29T11:33:25.294 INFO:teuthology.orchestra.run.smithi146.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 468 kB/s | 144 kB 00:00 2024-03-29T11:33:25.386 INFO:teuthology.orchestra.run.smithi146.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-29T11:33:25.561 INFO:teuthology.orchestra.run.smithi146.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.4 MB/s | 2.0 MB 00:00 2024-03-29T11:33:25.566 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:25.566 INFO:teuthology.orchestra.run.smithi146.stdout:Total 3.7 MB/s | 6.9 MB 00:01 2024-03-29T11:33:25.585 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:33:25.602 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:33:25.602 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:33:25.780 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:33:25.780 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:33:26.105 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:26.105 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:26.105 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:33:26.105 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:26.105 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:26.105 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:33:26.298 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:33:26.341 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr 2024-03-29T11:33:26.387 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:33:26.430 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:33:26.477 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:33:26.508 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:33:26.605 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:33:26.726 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:33:26.819 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:33:26.861 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:33:26.908 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:26.918 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:03 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:33:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:33:27.021 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:27.077 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:33:27.079 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-29T11:33:27.079 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:33:27.151 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:33:27.230 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:33:27.271 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:27.765 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:04 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:33:27.922 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:33:27.923 INFO:teuthology.orchestra.run.smithi037.stdout:====================================================================================== 2024-03-29T11:33:27.923 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-29T11:33:27.924 INFO:teuthology.orchestra.run.smithi037.stdout:====================================================================================== 2024-03-29T11:33:27.924 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:33:27.924 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:33:27.924 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-29T11:33:27.924 INFO:teuthology.orchestra.run.smithi037.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:33:27.924 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:33:27.925 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:33:27.925 INFO:teuthology.orchestra.run.smithi037.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:33:27.925 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:33:27.925 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:33:27.925 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:33:27.925 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:33:27.926 INFO:teuthology.orchestra.run.smithi037.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:33:27.926 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:33:27.926 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:33:27.926 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:27.926 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:33:27.926 INFO:teuthology.orchestra.run.smithi037.stdout:====================================================================================== 2024-03-29T11:33:27.927 INFO:teuthology.orchestra.run.smithi037.stdout:Install 12 Packages 2024-03-29T11:33:27.927 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:27.928 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 6.9 M 2024-03-29T11:33:27.928 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 26 M 2024-03-29T11:33:27.928 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:33:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:28.041 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:33:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:33:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:33:28.042 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:33:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:33:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:33:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:33:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:33:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:33:28.043 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:33:28.393 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:33:28.394 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:28.394 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:33:28.394 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:33:28.394 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:28.394 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:28.394 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:28.394 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:33:28.394 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:33:28.394 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:33:28.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:33:28.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:33:28.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:33:28.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:33:28.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:33:28.395 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:28.395 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:33:28.430 INFO:teuthology.orchestra.run.smithi037.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 82 kB/s | 15 kB 00:00 2024-03-29T11:33:28.505 INFO:teuthology.orchestra.run.smithi037.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 120 kB/s | 31 kB 00:00 2024-03-29T11:33:28.630 INFO:teuthology.orchestra.run.smithi037.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-29T11:33:28.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:33:28.656 INFO:teuthology.orchestra.run.smithi037.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-29T11:33:28.856 INFO:teuthology.orchestra.run.smithi037.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.9 MB/s | 1.8 MB 00:00 2024-03-29T11:33:28.881 INFO:teuthology.orchestra.run.smithi037.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.4 MB/s | 863 kB 00:00 2024-03-29T11:33:29.107 INFO:teuthology.orchestra.run.smithi037.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-03-29T11:33:29.206 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:33:29.265 INFO:teuthology.orchestra.run.smithi037.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 900 kB/s | 548 kB 00:00 2024-03-29T11:33:29.362 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:33:29.363 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-29T11:33:29.363 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:33:29.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:33:29.365 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:29.365 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:33:29.365 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-29T11:33:29.365 INFO:teuthology.orchestra.run.smithi146.stdout:Install 12 Packages 2024-03-29T11:33:29.365 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:29.365 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 37 M 2024-03-29T11:33:29.365 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 209 M 2024-03-29T11:33:29.366 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:33:29.382 INFO:teuthology.orchestra.run.smithi037.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 264 kB/s | 31 kB 00:00 2024-03-29T11:33:29.457 INFO:teuthology.orchestra.run.smithi037.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 412 kB/s | 144 kB 00:00 2024-03-29T11:33:29.499 INFO:teuthology.orchestra.run.smithi037.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-29T11:33:29.617 INFO:teuthology.orchestra.run.smithi146.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 188 kB/s | 32 kB 00:00 2024-03-29T11:33:29.692 INFO:teuthology.orchestra.run.smithi146.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 231 kB/s | 17 kB 00:00 2024-03-29T11:33:29.716 INFO:teuthology.orchestra.run.smithi037.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.4 MB/s | 2.0 MB 00:00 2024-03-29T11:33:29.720 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:29.720 INFO:teuthology.orchestra.run.smithi037.stdout:Total 3.8 MB/s | 6.9 MB 00:01 2024-03-29T11:33:29.740 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:33:29.759 INFO:teuthology.orchestra.run.smithi146.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 513 kB/s | 34 kB 00:00 2024-03-29T11:33:29.760 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:33:29.761 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:33:29.943 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:33:29.943 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:33:29.976 INFO:teuthology.orchestra.run.smithi146.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-03-29T11:33:30.126 INFO:teuthology.orchestra.run.smithi146.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-29T11:33:30.168 INFO:teuthology.orchestra.run.smithi146.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.1 MB/s | 217 kB 00:00 2024-03-29T11:33:30.467 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:33:30.551 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:33:30.594 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:33:30.602 INFO:teuthology.orchestra.run.smithi146.stdout:(7/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-29T11:33:30.641 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:33:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:33:30.781 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:33:30.827 INFO:teuthology.orchestra.run.smithi146.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-03-29T11:33:30.910 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:33:30.944 INFO:teuthology.orchestra.run.smithi146.stdout:(9/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.0 MB/s | 3.0 MB 00:01 2024-03-29T11:33:31.010 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:33:31.019 INFO:teuthology.orchestra.run.smithi146.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-29T11:33:31.059 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:33:31.086 INFO:teuthology.orchestra.run.smithi146.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-29T11:33:31.087 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:33:31.256 INFO:teuthology.orchestra.run.smithi146.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.1 MB/s | 7.4 MB 00:01 2024-03-29T11:33:31.259 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:31.259 INFO:teuthology.orchestra.run.smithi146.stdout:Total 20 MB/s | 37 MB 00:01 2024-03-29T11:33:31.299 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:33:31.323 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:33:31.323 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:33:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:33:31.443 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:31.695 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:33:31.695 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:33:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:33:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:33:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:33:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:33:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:33:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:33:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:33:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:33:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:33:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:33:32.408 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:33:32.476 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:33:32.505 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:33:32.505 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:32.505 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:33:32.506 INFO:teuthology.orchestra.run.smithi037.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:33:32.506 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:32.506 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:32.506 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:32.506 INFO:teuthology.orchestra.run.smithi037.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:33:32.506 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:33:32.506 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:33:32.506 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:33:32.506 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:33:32.506 INFO:teuthology.orchestra.run.smithi037.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:33:32.506 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:33:32.506 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:33:32.506 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:32.506 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:33:32.507 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:33:32.602 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:32.652 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:33:32.763 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:33:32.832 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:32.868 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:32.953 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:33:33.376 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:10 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:33:33.535 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:33:33.536 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================================ 2024-03-29T11:33:33.536 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-29T11:33:33.536 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================================ 2024-03-29T11:33:33.536 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:33:33.537 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:33:33.537 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-29T11:33:33.537 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:33:33.537 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:33:33.537 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:33:33.537 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:33:33.537 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:33:33.537 INFO:teuthology.orchestra.run.smithi037.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:33:33.537 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:33:33.537 INFO:teuthology.orchestra.run.smithi037.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:33:33.537 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:33:33.537 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:33:33.538 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:33:33.538 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:33.538 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:33:33.538 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================================ 2024-03-29T11:33:33.538 INFO:teuthology.orchestra.run.smithi037.stdout:Install 12 Packages 2024-03-29T11:33:33.538 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:33.538 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 37 M 2024-03-29T11:33:33.539 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 209 M 2024-03-29T11:33:33.539 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:33:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:33:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:33:33.947 INFO:teuthology.orchestra.run.smithi037.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 201 kB/s | 32 kB 00:00 2024-03-29T11:33:33.989 INFO:teuthology.orchestra.run.smithi037.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 428 kB/s | 17 kB 00:00 2024-03-29T11:33:34.039 INFO:teuthology.orchestra.run.smithi037.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 693 kB/s | 34 kB 00:00 2024-03-29T11:33:34.106 INFO:teuthology.orchestra.run.smithi037.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.4 MB/s | 3.0 MB 00:00 2024-03-29T11:33:34.340 INFO:teuthology.orchestra.run.smithi037.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2024-03-29T11:33:34.424 INFO:teuthology.orchestra.run.smithi037.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2024-03-29T11:33:34.448 INFO:teuthology.orchestra.run.smithi037.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-03-29T11:33:34.816 INFO:teuthology.orchestra.run.smithi037.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-29T11:33:35.008 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:33:35.358 INFO:teuthology.orchestra.run.smithi037.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 361 kB/s | 195 kB 00:00 2024-03-29T11:33:35.391 INFO:teuthology.orchestra.run.smithi037.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 854 kB/s | 803 kB 00:00 2024-03-29T11:33:35.458 INFO:teuthology.orchestra.run.smithi037.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-29T11:33:35.767 INFO:teuthology.orchestra.run.smithi037.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.7 MB/s | 7.4 MB 00:01 2024-03-29T11:33:35.769 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:35.769 INFO:teuthology.orchestra.run.smithi037.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-29T11:33:35.813 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:33:35.839 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:33:35.839 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:33:36.040 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:33:36.066 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:36.218 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:33:36.218 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:33:36.946 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:33:37.023 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:33:37.063 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:33:37.161 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:37.225 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:33:37.426 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:37.465 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:37.550 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:33:38.240 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:33:38.356 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:33:38.535 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:33:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:33:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:33:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:33:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:33:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:33:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:33:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:33:38.981 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:33:38.982 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:38.982 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:33:38.982 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:38.982 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:33:38.982 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:33:38.982 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:33:38.982 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:33:38.982 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:33:38.982 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:33:38.982 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:33:38.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:33:38.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:38.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:38.983 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:33:38.983 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:38.983 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:33:39.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:33:39.664 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:33:39.962 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:33:40.115 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:33:40.117 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-29T11:33:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-29T11:33:40.117 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-29T11:33:40.117 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:33:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:33:40.117 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-29T11:33:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:33:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:33:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:33:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:33:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:33:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:33:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:33:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:33:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:33:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:33:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:33:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:33:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:40.118 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:33:40.118 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-29T11:33:40.118 INFO:teuthology.orchestra.run.smithi146.stdout:Install 13 Packages 2024-03-29T11:33:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:40.119 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 2.0 M 2024-03-29T11:33:40.119 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 26 M 2024-03-29T11:33:40.119 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:33:40.345 INFO:teuthology.orchestra.run.smithi146.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 165 kB/s | 24 kB 00:00 2024-03-29T11:33:40.374 INFO:teuthology.orchestra.run.smithi146.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 102 kB/s | 18 kB 00:00 2024-03-29T11:33:40.433 INFO:teuthology.orchestra.run.smithi146.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-29T11:33:40.458 INFO:teuthology.orchestra.run.smithi146.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-29T11:33:40.484 INFO:teuthology.orchestra.run.smithi146.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:33:40.500 INFO:teuthology.orchestra.run.smithi146.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T11:33:40.584 INFO:teuthology.orchestra.run.smithi146.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 135 kB/s | 52 kB 00:00 2024-03-29T11:33:40.726 INFO:teuthology.orchestra.run.smithi146.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 66 kB/s | 15 kB 00:00 2024-03-29T11:33:40.732 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:33:40.764 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:40.776 INFO:teuthology.orchestra.run.smithi146.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-03-29T11:33:40.902 INFO:teuthology.orchestra.run.smithi146.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-03-29T11:33:40.977 INFO:teuthology.orchestra.run.smithi146.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 552 kB/s | 203 kB 00:00 2024-03-29T11:33:41.010 INFO:teuthology.orchestra.run.smithi146.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 828 kB/s | 90 kB 00:00 2024-03-29T11:33:41.069 INFO:teuthology.orchestra.run.smithi146.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2024-03-29T11:33:41.073 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:41.073 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-29T11:33:41.073 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:33:41.087 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:33:41.087 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:33:41.179 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:33:41.179 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:33:41.694 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:33:41.860 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:33:41.911 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:33:41.966 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:33:42.004 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:33:42.045 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:33:42.139 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:33:42.240 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:33:42.288 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:33:43.016 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:33:43.060 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:33:43.094 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:33:43.142 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:33:43.164 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:43.292 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:43.292 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:33:43.292 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:33:43.293 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:43.293 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:33:43.293 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:43.293 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:43.293 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:33:43.293 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:33:43.293 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:33:43.293 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:33:43.293 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:33:43.748 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:33:43.749 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:43.749 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:33:43.749 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:43.749 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:33:43.749 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:33:43.749 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:33:43.749 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:33:43.749 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:33:43.749 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:33:43.749 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:33:43.750 INFO:teuthology.orchestra.run.smithi037.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:33:43.750 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:43.750 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:43.750 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:33:43.750 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:43.750 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:33:44.016 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:33:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:33:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:33:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:33:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:33:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:33:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:33:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:33:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:33:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:33:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:33:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:33:44.105 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:33:44.403 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:33:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:44.503 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:33:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:33:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:33:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:33:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:33:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:33:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:33:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:33:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:33:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:33:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:33:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:33:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:33:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:44.504 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:33:44.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:33:44.682 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:33:44.840 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:33:44.841 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-29T11:33:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-29T11:33:44.842 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-29T11:33:44.842 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:33:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:33:44.842 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-29T11:33:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:33:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:33:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:33:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:33:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:33:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:33:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:33:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:33:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:33:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:33:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:33:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:33:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:44.843 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:33:44.843 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-29T11:33:44.843 INFO:teuthology.orchestra.run.smithi037.stdout:Install 13 Packages 2024-03-29T11:33:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:44.844 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 2.0 M 2024-03-29T11:33:44.844 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 26 M 2024-03-29T11:33:44.844 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:33:45.226 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:57 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:33:45.330 INFO:teuthology.orchestra.run.smithi037.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 153 kB/s | 18 kB 00:00 2024-03-29T11:33:45.346 INFO:teuthology.orchestra.run.smithi037.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 180 kB/s | 24 kB 00:00 2024-03-29T11:33:45.378 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:33:45.379 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-29T11:33:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-29T11:33:45.379 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-29T11:33:45.379 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:33:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:33:45.379 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-29T11:33:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:33:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:33:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:33:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:33:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:33:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:45.380 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:33:45.380 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-29T11:33:45.380 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2024-03-29T11:33:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:45.381 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 6.8 M 2024-03-29T11:33:45.381 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 33 M 2024-03-29T11:33:45.381 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:33:45.405 INFO:teuthology.orchestra.run.smithi037.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:33:45.430 INFO:teuthology.orchestra.run.smithi037.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-03-29T11:33:45.456 INFO:teuthology.orchestra.run.smithi037.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:33:45.481 INFO:teuthology.orchestra.run.smithi037.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 853 kB/s | 43 kB 00:00 2024-03-29T11:33:45.540 INFO:teuthology.orchestra.run.smithi037.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 159 kB/s | 52 kB 00:00 2024-03-29T11:33:45.656 INFO:teuthology.orchestra.run.smithi037.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 85 kB/s | 15 kB 00:00 2024-03-29T11:33:45.681 INFO:teuthology.orchestra.run.smithi037.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 143 kB/s | 32 kB 00:00 2024-03-29T11:33:45.807 INFO:teuthology.orchestra.run.smithi037.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-03-29T11:33:45.845 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-03-29T11:33:45.857 INFO:teuthology.orchestra.run.smithi037.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 639 kB/s | 203 kB 00:00 2024-03-29T11:33:45.920 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-29T11:33:45.924 INFO:teuthology.orchestra.run.smithi037.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 767 kB/s | 90 kB 00:00 2024-03-29T11:33:45.987 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 385 kB/s | 138 kB 00:00 2024-03-29T11:33:46.091 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 346 kB/s | 58 kB 00:00 2024-03-29T11:33:46.141 INFO:teuthology.orchestra.run.smithi037.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.1 MB/s | 1.0 MB 00:00 2024-03-29T11:33:46.143 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:46.143 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-29T11:33:46.144 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:33:46.145 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 322 kB/s | 51 kB 00:00 2024-03-29T11:33:46.159 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:33:46.159 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:33:46.179 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-03-29T11:33:46.181 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:46.181 INFO:teuthology.orchestra.run.smithi146.stdout:Total 8.5 MB/s | 6.8 MB 00:00 2024-03-29T11:33:46.195 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:33:46.205 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:33:46.205 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:33:46.260 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:33:46.262 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:33:46.295 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:33:46.295 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:33:46.611 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:33:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:33:46.782 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:33:46.839 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:33:46.953 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:33:47.018 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:33:47.083 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:33:47.118 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:33:47.160 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:33:47.258 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:33:47.350 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:33:47.367 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:33:47.420 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:33:47.433 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:33:47.495 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:33:47.518 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:48.200 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:33:48.250 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:33:48.283 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:33:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:33:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:33:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:33:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:33:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:33:48.341 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:33:48.370 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:48.616 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:33:48.616 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:48.617 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:33:48.617 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:48.617 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:33:48.617 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:33:48.617 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:33:48.617 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:33:48.617 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:33:48.617 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:48.617 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:33:48.861 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-fuse 2024-03-29T11:33:49.169 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:49.169 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:33:49.169 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:33:49.169 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:33:49.170 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:33:49.170 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:33:49.170 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:33:49.170 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:33:49.170 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:33:49.170 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:33:49.171 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:33:49.171 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:33:49.171 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:33:49.437 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:33:49.527 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:33:49.527 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:49.527 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:33:49.527 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:49.527 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:33:49.527 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:33:49.527 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:33:49.527 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:33:49.527 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:33:49.527 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:33:49.528 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:33:49.528 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:33:49.528 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:33:49.528 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:33:49.528 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:33:49.528 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:33:49.528 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:49.528 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:33:49.600 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:33:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:49.601 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-29T11:33:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:33:49.601 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:33:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-29T11:33:49.601 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:Install 2 Packages 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 964 k 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 2.8 M 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:33:49.780 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:33:49.838 INFO:teuthology.orchestra.run.smithi146.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 333 kB/s | 79 kB 00:00 2024-03-29T11:33:49.997 INFO:teuthology.orchestra.run.smithi146.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 885 kB 00:00 2024-03-29T11:33:49.998 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:49.999 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.4 MB/s | 964 kB 00:00 2024-03-29T11:33:49.999 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:33:50.019 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:33:50.019 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:33:50.092 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:33:50.092 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:33:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:33:50.357 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:33:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:33:50.489 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:50.515 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:33:50.516 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-29T11:33:50.516 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-29T11:33:50.516 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-29T11:33:50.516 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:33:50.516 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:33:50.516 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-29T11:33:50.516 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:33:50.516 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:33:50.516 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:33:50.517 INFO:teuthology.orchestra.run.smithi037.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:33:50.517 INFO:teuthology.orchestra.run.smithi037.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:33:50.517 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:50.517 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:33:50.517 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-29T11:33:50.517 INFO:teuthology.orchestra.run.smithi037.stdout:Install 6 Packages 2024-03-29T11:33:50.517 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:50.519 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 6.8 M 2024-03-29T11:33:50.519 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 33 M 2024-03-29T11:33:50.519 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:33:50.984 INFO:teuthology.orchestra.run.smithi037.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-29T11:33:51.176 INFO:teuthology.orchestra.run.smithi037.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-03-29T11:33:51.186 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:51.186 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:33:51.234 INFO:teuthology.orchestra.run.smithi037.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 328 kB/s | 138 kB 00:00 2024-03-29T11:33:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:33:51.500 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:51.501 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:33:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:33:51.501 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:51.501 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:33:51.610 INFO:teuthology.orchestra.run.smithi037.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 133 kB/s | 58 kB 00:00 2024-03-29T11:33:51.659 INFO:teuthology.orchestra.run.smithi037.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 120 kB/s | 51 kB 00:00 2024-03-29T11:33:51.685 INFO:teuthology.orchestra.run.smithi037.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 783 kB/s | 548 kB 00:00 2024-03-29T11:33:51.689 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:51.689 INFO:teuthology.orchestra.run.smithi037.stdout:Total 5.8 MB/s | 6.8 MB 00:01 2024-03-29T11:33:51.702 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:33:51.714 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:33:51.714 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:33:51.801 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-volume 2024-03-29T11:33:51.808 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:33:51.810 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:33:52.105 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:33:52.267 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:33:52.321 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:33:52.372 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:04 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:33:52.531 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:33:52.532 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-29T11:33:52.532 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:52.532 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:33:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:33:52.532 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-29T11:33:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:33:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:52.532 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:33:52.533 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:52.533 INFO:teuthology.orchestra.run.smithi146.stdout:Install 2 Packages 2024-03-29T11:33:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:52.533 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 332 k 2024-03-29T11:33:52.533 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 1.5 M 2024-03-29T11:33:52.533 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:33:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 585 kB/s | 69 kB 00:00 2024-03-29T11:33:52.868 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:33:52.943 INFO:teuthology.orchestra.run.smithi146.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 644 kB/s | 264 kB 00:00 2024-03-29T11:33:52.944 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:52.944 INFO:teuthology.orchestra.run.smithi146.stdout:Total 807 kB/s | 332 kB 00:00 2024-03-29T11:33:52.945 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:33:52.953 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:33:52.953 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:33:52.958 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:33:52.992 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:33:52.992 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:33:53.027 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:33:53.065 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:53.137 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:33:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:33:53.354 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:53.370 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:53.370 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:53.370 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:33:53.370 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:53.840 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:53.840 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:33:53.840 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:33:53.840 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:33:53.840 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:33:53.840 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:33:54.150 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:33:54.150 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:54.151 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:33:54.151 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:54.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:33:54.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:33:54.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:33:54.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:33:54.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:33:54.151 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:54.151 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:33:54.238 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:33:54.403 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-fuse 2024-03-29T11:33:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:33:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:54.586 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:33:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:33:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:54.586 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:33:54.910 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados-devel 2024-03-29T11:33:54.993 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:31 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:33:55.152 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:33:55.152 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:33:55.153 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-29T11:33:55.153 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:33:55.153 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:33:55.153 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:33:55.153 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-29T11:33:55.153 INFO:teuthology.orchestra.run.smithi037.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:33:55.153 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:55.153 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:33:55.153 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:33:55.153 INFO:teuthology.orchestra.run.smithi037.stdout:Install 2 Packages 2024-03-29T11:33:55.153 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:55.153 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 964 k 2024-03-29T11:33:55.153 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 2.8 M 2024-03-29T11:33:55.153 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:33:55.287 INFO:teuthology.orchestra.run.smithi037.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 594 kB/s | 79 kB 00:00 2024-03-29T11:33:55.478 INFO:teuthology.orchestra.run.smithi037.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.7 MB/s | 885 kB 00:00 2024-03-29T11:33:55.480 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:55.480 INFO:teuthology.orchestra.run.smithi037.stdout:Total 2.9 MB/s | 964 kB 00:00 2024-03-29T11:33:55.480 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:33:55.489 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:07 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:33:55.501 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:33:55.501 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:33:55.579 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:33:55.579 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:33:55.640 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:33:55.641 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-29T11:33:55.641 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:55.641 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:33:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:33:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:55.641 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:33:55.641 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:55.641 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-29T11:33:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:55.641 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 127 k 2024-03-29T11:33:55.641 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 456 k 2024-03-29T11:33:55.641 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:33:55.769 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:33:55.917 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:33:55.982 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:55.990 INFO:teuthology.orchestra.run.smithi146.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 363 kB/s | 127 kB 00:00 2024-03-29T11:33:55.991 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:55.991 INFO:teuthology.orchestra.run.smithi146.stdout:Total 361 kB/s | 127 kB 00:00 2024-03-29T11:33:55.992 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:33:55.997 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:33:55.997 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:33:56.053 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:33:56.053 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:33:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:33:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:33:56.761 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:57.025 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:33:57.025 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:57.025 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:33:57.025 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:33:57.026 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:57.026 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:33:57.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:57.057 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:57.057 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:33:57.058 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:57.058 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:57.058 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:33:57.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs2 2024-03-29T11:33:57.292 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-volume 2024-03-29T11:33:57.859 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:09 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:33:57.872 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:34 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:33:57.963 INFO:teuthology.orchestra.run.smithi146.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:58.017 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:33:58.018 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-29T11:33:58.018 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:33:58.032 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:33:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:33:58.033 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-29T11:33:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:33:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:33:58.033 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:33:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-29T11:33:58.033 INFO:teuthology.orchestra.run.smithi037.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:33:58.033 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:33:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:33:58.033 INFO:teuthology.orchestra.run.smithi037.stdout:Install 2 Packages 2024-03-29T11:33:58.033 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 332 k 2024-03-29T11:33:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 1.5 M 2024-03-29T11:33:58.034 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:33:58.078 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs-devel 2024-03-29T11:33:58.151 INFO:teuthology.orchestra.run.smithi037.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 590 kB/s | 69 kB 00:00 2024-03-29T11:33:58.435 INFO:teuthology.orchestra.run.smithi037.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 659 kB/s | 264 kB 00:00 2024-03-29T11:33:58.436 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:58.436 INFO:teuthology.orchestra.run.smithi037.stdout:Total 825 kB/s | 332 kB 00:00 2024-03-29T11:33:58.436 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:33:58.446 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:33:58.446 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:33:58.484 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:33:58.485 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:33:58.635 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:33:58.649 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:10 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:33:58.804 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:33:58.805 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-29T11:33:58.805 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:58.805 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:33:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:33:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:58.805 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:33:58.805 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:33:58.805 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-29T11:33:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:58.805 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 31 k 2024-03-29T11:33:58.805 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 140 k 2024-03-29T11:33:58.806 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:33:58.823 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:33:58.854 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:58.869 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:58.869 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:58.869 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:33:58.869 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:59.001 INFO:teuthology.orchestra.run.smithi146.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 159 kB/s | 31 kB 00:00 2024-03-29T11:33:59.002 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:59.002 INFO:teuthology.orchestra.run.smithi146.stdout:Total 158 kB/s | 31 kB 00:00 2024-03-29T11:33:59.002 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:33:59.005 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:33:59.005 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:33:59.034 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:33:59.035 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:33:59.146 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:33:59.275 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:59.538 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:33:59.686 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:59.849 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:33:59.849 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:59.849 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:33:59.849 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:59.849 INFO:teuthology.orchestra.run.smithi037.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:33:59.850 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:33:59.850 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:33:59.967 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:59.967 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:59.967 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:33:59.967 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:59.967 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:33:59.967 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:34:00.120 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados2 2024-03-29T11:34:00.193 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados-devel 2024-03-29T11:34:00.686 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:12 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:34:00.773 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:37 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:34:00.786 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:00.837 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:34:00.838 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-29T11:34:00.838 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:34:00.890 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librbd1 2024-03-29T11:34:00.929 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:34:00.929 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:00.929 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-29T11:34:00.929 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:00.930 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:34:00.930 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:34:00.930 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:00.930 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:34:00.930 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:00.930 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-29T11:34:00.930 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:00.931 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 127 k 2024-03-29T11:34:00.931 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 456 k 2024-03-29T11:34:00.931 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:34:01.132 INFO:teuthology.orchestra.run.smithi037.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 631 kB/s | 127 kB 00:00 2024-03-29T11:34:01.132 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:01.132 INFO:teuthology.orchestra.run.smithi037.stdout:Total 627 kB/s | 127 kB 00:00 2024-03-29T11:34:01.132 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:34:01.138 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:34:01.138 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:34:01.196 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:34:01.196 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:34:01.339 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:34:01.441 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:01.454 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:13 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:34:01.552 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:01.604 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:34:01.605 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-29T11:34:01.605 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:34:01.651 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rados 2024-03-29T11:34:01.885 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:02.183 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:02.183 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:02.183 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:34:02.183 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:02.183 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:02.183 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:34:02.217 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:14 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:34:02.316 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:02.368 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:34:02.369 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-29T11:34:02.369 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:34:02.411 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rgw 2024-03-29T11:34:02.427 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs2 2024-03-29T11:34:02.978 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:14 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:34:03.009 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:40 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:34:03.076 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:03.112 INFO:teuthology.orchestra.run.smithi037.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:03.128 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:34:03.129 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-29T11:34:03.129 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:34:03.168 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:34:03.169 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-29T11:34:03.169 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:34:03.176 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-cephfs 2024-03-29T11:34:03.247 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs-devel 2024-03-29T11:34:03.749 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:34:03.825 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:40 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:34:03.847 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:03.900 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:34:03.901 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-29T11:34:03.901 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:34:03.948 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rbd 2024-03-29T11:34:03.982 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:34:03.983 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:03.983 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-29T11:34:03.983 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:03.983 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:34:03.983 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:34:03.983 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:03.983 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:34:03.983 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:03.983 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-29T11:34:03.983 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:03.984 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 31 k 2024-03-29T11:34:03.984 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 140 k 2024-03-29T11:34:03.984 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:34:04.151 INFO:teuthology.orchestra.run.smithi037.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 187 kB/s | 31 kB 00:00 2024-03-29T11:34:04.153 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:04.153 INFO:teuthology.orchestra.run.smithi037.stdout:Total 185 kB/s | 31 kB 00:00 2024-03-29T11:34:04.153 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:34:04.156 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:34:04.156 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:34:04.197 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:34:04.197 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:34:04.333 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:34:04.452 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:04.527 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:16 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:34:04.626 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:04.678 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:34:04.679 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-29T11:34:04.679 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:34:04.724 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-fuse 2024-03-29T11:34:04.820 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:05.093 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:05.093 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:05.093 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:34:05.093 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:05.093 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:05.093 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:34:05.278 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados2 2024-03-29T11:34:05.300 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:17 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:34:05.450 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:34:05.451 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:34:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-03-29T11:34:05.451 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:34:05.451 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:34:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:34:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:05.451 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:34:05.451 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:34:05.451 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-29T11:34:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:05.451 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 85 k 2024-03-29T11:34:05.452 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 231 k 2024-03-29T11:34:05.452 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:34:05.662 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 405 kB/s | 85 kB 00:00 2024-03-29T11:34:05.663 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:05.663 INFO:teuthology.orchestra.run.smithi146.stdout:Total 402 kB/s | 85 kB 00:00 2024-03-29T11:34:05.663 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:34:05.670 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:34:05.670 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:34:05.725 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:34:05.726 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:34:05.848 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:42 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:34:05.870 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:34:05.947 INFO:teuthology.orchestra.run.smithi037.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:05.969 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:06.002 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:34:06.003 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-29T11:34:06.003 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:34:06.051 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librbd1 2024-03-29T11:34:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:06.622 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:43 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:34:06.681 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:06.681 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:06.681 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:34:06.681 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:06.681 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:06.681 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:34:06.722 INFO:teuthology.orchestra.run.smithi037.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:06.777 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:34:06.778 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-29T11:34:06.778 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:34:06.830 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rados 2024-03-29T11:34:06.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-mirror 2024-03-29T11:34:07.412 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:44 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:34:07.478 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:34:07.516 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:07.570 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:34:07.572 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-29T11:34:07.572 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:34:07.622 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rgw 2024-03-29T11:34:07.632 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:34:07.632 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:34:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-29T11:34:07.633 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:34:07.633 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:34:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:34:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:07.633 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:34:07.633 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:34:07.633 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-29T11:34:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:07.634 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 3.1 M 2024-03-29T11:34:07.634 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 13 M 2024-03-29T11:34:07.634 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:34:08.191 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.5 MB/s | 3.1 MB 00:00 2024-03-29T11:34:08.194 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:08.194 INFO:teuthology.orchestra.run.smithi146.stdout:Total 5.5 MB/s | 3.1 MB 00:00 2024-03-29T11:34:08.194 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:34:08.196 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:45 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:34:08.202 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:34:08.202 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:34:08.274 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:34:08.275 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:34:08.298 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:08.354 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:34:08.355 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-29T11:34:08.355 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:34:08.405 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-cephfs 2024-03-29T11:34:08.701 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:34:08.759 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:08.783 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:08.784 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:34:08.784 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:34:08.784 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:34:08.784 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:34:08.784 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:08.986 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:45 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:34:09.087 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:09.142 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:34:09.143 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-29T11:34:09.143 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:34:09.193 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rbd 2024-03-29T11:34:09.770 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:46 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:34:09.872 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:09.928 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:34:09.929 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-29T11:34:09.929 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:34:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:09.947 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:34:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:09.947 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:09.947 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:34:09.978 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-fuse 2024-03-29T11:34:10.193 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-nbd 2024-03-29T11:34:10.546 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:47 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:34:10.701 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:34:10.701 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:10.701 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-03-29T11:34:10.701 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:10.701 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:34:10.701 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:34:10.702 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:10.702 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:34:10.702 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:10.702 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-29T11:34:10.702 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:10.702 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 85 k 2024-03-29T11:34:10.702 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 231 k 2024-03-29T11:34:10.703 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:34:10.770 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:22 ago on Fri 29 Mar 2024 11:31:48 AM UTC. 2024-03-29T11:34:10.921 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-29T11:34:10.922 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:34:10.922 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-03-29T11:34:10.922 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:34:10.922 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-29T11:34:10.923 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:34:10.923 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:10.923 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-29T11:34:10.923 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-29T11:34:10.923 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-29T11:34:10.923 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:10.923 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 171 k 2024-03-29T11:34:10.923 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 494 k 2024-03-29T11:34:10.923 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-29T11:34:11.021 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 268 kB/s | 85 kB 00:00 2024-03-29T11:34:11.021 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:11.021 INFO:teuthology.orchestra.run.smithi037.stdout:Total 267 kB/s | 85 kB 00:00 2024-03-29T11:34:11.022 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:34:11.029 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:34:11.029 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:34:11.087 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:34:11.087 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:34:11.144 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 776 kB/s | 171 kB 00:00 2024-03-29T11:34:11.145 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:11.145 INFO:teuthology.orchestra.run.smithi146.stdout:Total 771 kB/s | 171 kB 00:00 2024-03-29T11:34:11.145 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-29T11:34:11.153 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-29T11:34:11.153 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-29T11:34:11.212 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-29T11:34:11.212 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-29T11:34:11.226 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:34:11.323 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:11.362 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-29T11:34:11.459 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:11.930 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:12.032 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:12.032 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:12.032 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:34:12.032 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:12.032 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:12.033 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:34:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:12.219 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-29T11:34:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:12.219 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-29T11:34:12.263 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-mirror 2024-03-29T11:34:12.465 DEBUG:teuthology.parallel:result is None 2024-03-29T11:34:12.850 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:49 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:34:13.010 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:34:13.010 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:13.010 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-29T11:34:13.010 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:13.010 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:34:13.011 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:34:13.011 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:13.011 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:34:13.011 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:13.011 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-29T11:34:13.011 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:13.011 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 3.1 M 2024-03-29T11:34:13.011 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 13 M 2024-03-29T11:34:13.011 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:34:13.873 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 3.6 MB/s | 3.1 MB 00:00 2024-03-29T11:34:13.874 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:13.874 INFO:teuthology.orchestra.run.smithi037.stdout:Total 3.6 MB/s | 3.1 MB 00:00 2024-03-29T11:34:13.875 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:34:13.885 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:34:13.885 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:34:13.962 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:34:13.962 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:34:14.410 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:34:14.474 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:14.498 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:14.499 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:34:14.499 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:34:14.499 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:34:14.499 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:34:14.499 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:15.583 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:34:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:15.583 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:34:15.843 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-nbd 2024-03-29T11:34:16.422 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:53 ago on Fri 29 Mar 2024 11:31:23 AM UTC. 2024-03-29T11:34:16.576 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-29T11:34:16.577 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:16.577 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-03-29T11:34:16.577 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:16.577 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-29T11:34:16.577 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:34:16.577 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:16.578 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-29T11:34:16.578 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-29T11:34:16.578 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-29T11:34:16.578 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:16.578 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 171 k 2024-03-29T11:34:16.578 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 494 k 2024-03-29T11:34:16.578 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-29T11:34:16.801 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 769 kB/s | 171 kB 00:00 2024-03-29T11:34:16.802 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:16.802 INFO:teuthology.orchestra.run.smithi037.stdout:Total 764 kB/s | 171 kB 00:00 2024-03-29T11:34:16.802 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-29T11:34:16.811 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-29T11:34:16.811 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-29T11:34:16.870 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-29T11:34:16.870 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-29T11:34:17.023 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-29T11:34:17.127 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:17.577 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:17.864 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:17.864 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:17.864 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-29T11:34:17.864 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:17.864 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:17.864 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-29T11:34:18.088 DEBUG:teuthology.parallel:result is None 2024-03-29T11:34:18.088 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:34:18.230 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:34:18.250 INFO:teuthology.orchestra.run.smithi037.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:34:18.250 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:34:18.250 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:34:18.252 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:34:18.474 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:34:18.499 INFO:teuthology.orchestra.run.smithi146.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:34:18.500 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:34:18.500 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:34:18.502 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:34:18.502 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:18.502 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:34:18.530 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:18.530 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:34:18.571 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:34:18.571 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:18.571 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:34:18.597 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:34:18.664 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:18.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:34:18.694 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:34:18.764 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:34:18.764 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:18.764 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:34:18.790 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:34:18.857 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:18.857 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:34:18.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:34:18.955 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:34:18.955 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:18.955 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:34:18.982 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:34:19.049 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:19.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:34:19.077 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:34:19.145 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:34:19.230 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:34:19.230 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 777 /var/log/ceph 2024-03-29T11:34:19.232 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /var/log/ceph 2024-03-29T11:34:19.260 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:34:19.260 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:34:19.299 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:34:19.328 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:34:19.329 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:34:19.365 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:34:19.406 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:34:19.406 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:34:19.406 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622721', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi037.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLoffezH04GjNximqgvmhGPgsmhf2UKf/l7edPWBk4Yd7DA6Ai5XwXMG3gQT7qxKm1AO8CW3cwfU6suMj4+pL/Y=', 'smithi146.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOgITzAaFXRUJPNt5KLN+jaof4PCZTceu09kE0/gT7yB+Qt2bdVMpg+Ey3GHI4/SO7sl5/Lw3mygW0ryf7Gz3BY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:34:19.407 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:34:19.432 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:34:19.451 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:34:19.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:34:19.518 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:19.518 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:34:19.565 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:34:19.565 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_1 2024-03-29T11:34:19.620 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:34:19.620 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:19.620 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:34:19.620 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:19.620 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:19.620 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-03-29 11:34:14.757548594 +0000 2024-03-29T11:34:19.620 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-03-29 11:30:41.344516258 +0000 2024-03-29T11:34:19.620 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-03-29 11:30:41.344516258 +0000 2024-03-29T11:34:19.620 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-03-29 11:30:41.344516258 +0000 2024-03-29T11:34:19.620 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:34:19.684 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-03-29T11:34:19.684 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-03-29T11:34:19.684 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000244833 s, 2.1 MB/s 2024-03-29T11:34:19.685 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:34:19.741 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_2 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-03-29 11:34:14.757548594 +0000 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-03-29 11:30:41.697511714 +0000 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-03-29 11:30:41.697511714 +0000 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-03-29 11:30:41.697511714 +0000 2024-03-29T11:34:19.795 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:34:19.859 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-03-29T11:34:19.859 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-03-29T11:34:19.859 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000201123 s, 2.5 MB/s 2024-03-29T11:34:19.860 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:34:19.916 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_3 2024-03-29T11:34:19.970 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:34:19.970 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:19.970 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:34:19.970 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:19.970 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:19.970 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-03-29 11:34:14.757548594 +0000 2024-03-29T11:34:19.970 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-03-29 11:30:42.038507324 +0000 2024-03-29T11:34:19.970 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-03-29 11:30:42.038507324 +0000 2024-03-29T11:34:19.970 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-03-29 11:30:42.038507324 +0000 2024-03-29T11:34:19.970 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:34:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-03-29T11:34:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-03-29T11:34:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000192665 s, 2.7 MB/s 2024-03-29T11:34:20.036 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:34:20.090 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_4 2024-03-29T11:34:20.145 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:34:20.145 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:20.145 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T11:34:20.145 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:20.145 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:20.145 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-03-29 11:34:14.758548581 +0000 2024-03-29T11:34:20.145 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-03-29 11:30:42.395502728 +0000 2024-03-29T11:34:20.145 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-03-29 11:30:42.395502728 +0000 2024-03-29T11:34:20.145 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-03-29 11:30:42.395502728 +0000 2024-03-29T11:34:20.145 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:34:20.209 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-03-29T11:34:20.209 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-03-29T11:34:20.209 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000193937 s, 2.6 MB/s 2024-03-29T11:34:20.210 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:34:20.266 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:20.266 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:20.266 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:34:20.283 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:34:20.283 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-03-29T11:34:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:34:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:20.340 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T11:34:20.340 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:20.341 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:20.341 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-29 11:34:09.043010816 +0000 2024-03-29T11:34:20.341 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-29 11:30:38.858988880 +0000 2024-03-29T11:34:20.341 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-29 11:30:38.858988880 +0000 2024-03-29T11:34:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-29 11:30:38.858988880 +0000 2024-03-29T11:34:20.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:34:20.409 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-29T11:34:20.409 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-29T11:34:20.409 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000217111 s, 2.4 MB/s 2024-03-29T11:34:20.410 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:34:20.470 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-03-29T11:34:20.527 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:34:20.527 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:20.527 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T11:34:20.527 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:20.528 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:20.528 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-29 11:34:09.043010816 +0000 2024-03-29T11:34:20.528 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-29 11:30:39.212983069 +0000 2024-03-29T11:34:20.528 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-29 11:30:39.212983069 +0000 2024-03-29T11:34:20.528 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-29 11:30:39.212983069 +0000 2024-03-29T11:34:20.528 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:34:20.596 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-29T11:34:20.596 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-29T11:34:20.596 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000198089 s, 2.6 MB/s 2024-03-29T11:34:20.597 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:34:20.655 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-03-29T11:34:20.710 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:34:20.711 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:20.711 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:34:20.711 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:20.711 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:20.711 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-29 11:34:09.044010800 +0000 2024-03-29T11:34:20.711 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-29 11:30:39.559977372 +0000 2024-03-29T11:34:20.711 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-29 11:30:39.559977372 +0000 2024-03-29T11:34:20.711 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-29 11:30:39.559977372 +0000 2024-03-29T11:34:20.711 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:34:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-29T11:34:20.778 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-29T11:34:20.778 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000226911 s, 2.3 MB/s 2024-03-29T11:34:20.779 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:34:20.835 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-03-29T11:34:20.891 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:34:20.891 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:20.891 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T11:34:20.891 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:20.891 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:20.891 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-29 11:34:09.044010800 +0000 2024-03-29T11:34:20.891 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-29 11:30:39.918971478 +0000 2024-03-29T11:34:20.891 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-29 11:30:39.918971478 +0000 2024-03-29T11:34:20.891 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-29 11:30:39.918971478 +0000 2024-03-29T11:34:20.892 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:34:20.957 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-29T11:34:20.957 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-29T11:34:20.957 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000191575 s, 2.7 MB/s 2024-03-29T11:34:20.958 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:34:21.014 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:21.014 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:34:21.014 INFO:tasks.ceph:Generating config... 2024-03-29T11:34:21.016 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:34:21.016 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:34:21.016 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:34:21.016 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:34:21.016 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:34:21.016 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:34:21.016 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:34:21.016 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:34:21.016 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:34:21.016 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:34:21.016 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:34:21.017 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:34:21.017 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:34:21.017 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:34:21.017 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:34:21.017 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:34:21.017 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:34:21.017 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:34:21.017 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:34:21.017 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:34:21.017 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:34:21.017 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:34:21.017 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:34:21.017 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:34:21.017 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:34:21.018 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:34:21.018 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:34:21.018 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:34:21.018 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:34:21.018 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:34:21.018 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:34:21.018 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:34:21.018 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:34:21.018 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:34:21.018 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:34:21.018 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:34:21.018 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:34:21.018 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:34:21.019 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:34:21.019 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:34:21.019 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:34:21.019 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:34:21.019 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:34:21.019 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:34:21.019 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:34:21.019 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:34:21.019 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:34:21.019 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:34:21.019 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:34:21.019 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:21.074 INFO:teuthology.orchestra.run.smithi037.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:34:21.077 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:34:21.123 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:34:21.150 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.37:6789]'), ('mon.c', '[v1:172.21.15.37:6790]'), ('mon.b', '[v1:172.21.15.146:6789]')] 2024-03-29T11:34:21.151 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.37:6789],[v1:172.21.15.37:6790],[v1:172.21.15.146:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:34:21.151 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:21.152 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:34:21.206 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.37:6789]' --addv c '[v1:172.21.15.37:6790]' --addv b '[v1:172.21.15.146:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:21.290 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:21.291 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: generated fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:34:21.291 INFO:teuthology.orchestra.run.smithi037.stdout:setting min_mon_release = quincy 2024-03-29T11:34:21.291 INFO:teuthology.orchestra.run.smithi037.stdout:epoch 0 2024-03-29T11:34:21.291 INFO:teuthology.orchestra.run.smithi037.stdout:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:34:21.294 INFO:teuthology.orchestra.run.smithi037.stdout:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:34:21.294 INFO:teuthology.orchestra.run.smithi037.stdout:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:34:21.294 INFO:teuthology.orchestra.run.smithi037.stdout:min_mon_release 17 (quincy) 2024-03-29T11:34:21.294 INFO:teuthology.orchestra.run.smithi037.stdout:election_strategy: 1 2024-03-29T11:34:21.294 INFO:teuthology.orchestra.run.smithi037.stdout:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:34:21.294 INFO:teuthology.orchestra.run.smithi037.stdout:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:34:21.294 INFO:teuthology.orchestra.run.smithi037.stdout:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:34:21.295 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:34:21.296 DEBUG:teuthology.orchestra.run.smithi037:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:34:21.311 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5a9df19a-9269-4a6f-b895-975ff91e9fb9... 2024-03-29T11:34:21.312 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:34:21.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:34:21.402 INFO:teuthology.orchestra.run.smithi037.stdout:[global] 2024-03-29T11:34:21.402 INFO:teuthology.orchestra.run.smithi037.stdout: chdir = "" 2024-03-29T11:34:21.402 INFO:teuthology.orchestra.run.smithi037.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:34:21.403 INFO:teuthology.orchestra.run.smithi037.stdout: auth supported = cephx 2024-03-29T11:34:21.403 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.403 INFO:teuthology.orchestra.run.smithi037.stdout: filestore xattr use omap = true 2024-03-29T11:34:21.403 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.403 INFO:teuthology.orchestra.run.smithi037.stdout: mon clock drift allowed = 1.000 2024-03-29T11:34:21.403 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.403 INFO:teuthology.orchestra.run.smithi037.stdout: osd crush chooseleaf type = 0 2024-03-29T11:34:21.403 INFO:teuthology.orchestra.run.smithi037.stdout: auth debug = true 2024-03-29T11:34:21.403 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.403 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on old message = true 2024-03-29T11:34:21.403 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on bug = true 2024-03-29T11:34:21.403 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.404 INFO:teuthology.orchestra.run.smithi037.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:34:21.404 INFO:teuthology.orchestra.run.smithi037.stdout: mon pg warn max object skew = 0 2024-03-29T11:34:21.404 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.404 INFO:teuthology.orchestra.run.smithi037.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:34:21.404 INFO:teuthology.orchestra.run.smithi037.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:34:21.404 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.404 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default size = 2 2024-03-29T11:34:21.404 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.404 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow primary affinity = true 2024-03-29T11:34:21.404 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow pg remap = true 2024-03-29T11:34:21.404 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:34:21.404 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:34:21.404 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on no sortbitwise = false 2024-03-29T11:34:21.405 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:34:21.405 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on too few osds = false 2024-03-29T11:34:21.405 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:34:21.405 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:34:21.405 INFO:teuthology.orchestra.run.smithi037.stdout: mon_allow_pool_size_one = true 2024-03-29T11:34:21.405 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.405 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:34:21.405 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.405 INFO:teuthology.orchestra.run.smithi037.stdout: osd default data pool replay window = 5 2024-03-29T11:34:21.405 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.405 INFO:teuthology.orchestra.run.smithi037.stdout: mon allow pool delete = true 2024-03-29T11:34:21.405 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.405 INFO:teuthology.orchestra.run.smithi037.stdout: mon cluster log file level = debug 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout: debug asserts on shutdown = true 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout: mon health detail to clog = false 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout: mon host = "[v1:172.21.15.37:6789],[v1:172.21.15.37:6790],[v1:172.21.15.146:6789]" 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout: lockdep = True 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout: mon client directed command retry = 5 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout: mon election default strategy = 1 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout: ms bind msgr2 = False 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject socket failures = 5000 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout: ms type = async 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout: fsid = 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout:[osd] 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout: osd journal size = 100 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.406 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub load threshold = 5.0 2024-03-29T11:34:21.407 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub max interval = 600 2024-03-29T11:34:21.407 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:34:21.407 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.407 INFO:teuthology.orchestra.run.smithi037.stdout: osd recover clone overlap = true 2024-03-29T11:34:21.407 INFO:teuthology.orchestra.run.smithi037.stdout: osd recovery max chunk = 1048576 2024-03-29T11:34:21.407 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.407 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug shutdown = true 2024-03-29T11:34:21.407 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug op order = true 2024-03-29T11:34:21.407 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify stray on activate = true 2024-03-29T11:34:21.407 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug trim objects = true 2024-03-29T11:34:21.407 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.407 INFO:teuthology.orchestra.run.smithi037.stdout: osd open classes on start = true 2024-03-29T11:34:21.407 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug pg log writeout = true 2024-03-29T11:34:21.407 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.408 INFO:teuthology.orchestra.run.smithi037.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:34:21.408 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.408 INFO:teuthology.orchestra.run.smithi037.stdout: osd map max advance = 10 2024-03-29T11:34:21.408 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.408 INFO:teuthology.orchestra.run.smithi037.stdout: journal zero on create = true 2024-03-29T11:34:21.408 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.408 INFO:teuthology.orchestra.run.smithi037.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:34:21.408 INFO:teuthology.orchestra.run.smithi037.stdout: filestore apply finisher threads = 3 2024-03-29T11:34:21.408 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.408 INFO:teuthology.orchestra.run.smithi037.stdout: bdev debug aio = true 2024-03-29T11:34:21.408 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug misdirected ops = true 2024-03-29T11:34:21.408 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore block size = 96636764160 2024-03-29T11:34:21.408 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore compression algorithm = snappy 2024-03-29T11:34:21.408 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore compression mode = aggressive 2024-03-29T11:34:21.409 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore fsck on mount = True 2024-03-29T11:34:21.409 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore zero block detection = True 2024-03-29T11:34:21.409 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluefs = 20 2024-03-29T11:34:21.409 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluestore = 20 2024-03-29T11:34:21.409 INFO:teuthology.orchestra.run.smithi037.stdout: debug monc = 20 2024-03-29T11:34:21.409 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-03-29T11:34:21.409 INFO:teuthology.orchestra.run.smithi037.stdout: debug osd = 20 2024-03-29T11:34:21.409 INFO:teuthology.orchestra.run.smithi037.stdout: debug rocksdb = 10 2024-03-29T11:34:21.409 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:34:21.410 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd full ratio = 0.9 2024-03-29T11:34:21.410 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:34:21.410 INFO:teuthology.orchestra.run.smithi037.stdout: osd class default list = * 2024-03-29T11:34:21.410 INFO:teuthology.orchestra.run.smithi037.stdout: osd class load list = * 2024-03-29T11:34:21.410 INFO:teuthology.orchestra.run.smithi037.stdout: osd client watch timeout = 120 2024-03-29T11:34:21.410 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify cached snaps = True 2024-03-29T11:34:21.410 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify missing on start = True 2024-03-29T11:34:21.410 INFO:teuthology.orchestra.run.smithi037.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:34:21.410 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock override recovery settings = True 2024-03-29T11:34:21.411 INFO:teuthology.orchestra.run.smithi037.stdout: osd objectstore = bluestore 2024-03-29T11:34:21.411 INFO:teuthology.orchestra.run.smithi037.stdout: osd op queue = debug_random 2024-03-29T11:34:21.411 INFO:teuthology.orchestra.run.smithi037.stdout: osd op queue cut off = debug_random 2024-03-29T11:34:21.411 INFO:teuthology.orchestra.run.smithi037.stdout: osd shutdown pgref assert = True 2024-03-29T11:34:21.411 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.411 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr] 2024-03-29T11:34:21.411 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-03-29T11:34:21.411 INFO:teuthology.orchestra.run.smithi037.stdout: debug mgr = 20 2024-03-29T11:34:21.411 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-03-29T11:34:21.411 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-03-29T11:34:21.411 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:21.411 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:21.411 INFO:teuthology.orchestra.run.smithi037.stdout: mgr/telemetry/nag = false 2024-03-29T11:34:21.412 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.412 INFO:teuthology.orchestra.run.smithi037.stdout:[mon] 2024-03-29T11:34:21.412 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-03-29T11:34:21.412 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-03-29T11:34:21.412 INFO:teuthology.orchestra.run.smithi037.stdout: debug paxos = 20 2024-03-29T11:34:21.412 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-03-29T11:34:21.412 INFO:teuthology.orchestra.run.smithi037.stdout: mon data avail warn = 5 2024-03-29T11:34:21.412 INFO:teuthology.orchestra.run.smithi037.stdout: mon mgr mkfs grace = 240 2024-03-29T11:34:21.412 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:21.412 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd reporter subtree level = osd 2024-03-29T11:34:21.412 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd prime pg temp = true 2024-03-29T11:34:21.412 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:21.412 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.412 INFO:teuthology.orchestra.run.smithi037.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: # 1m isn't quite enough 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: mon min osdmap epochs = 50 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune interval = 2 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune min = 15 2024-03-29T11:34:21.413 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: mon scrub interval = 300 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on pool no app = False 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: paxos service trim min = 10 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout:[client] 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: rgw cache enabled = true 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable ops log = true 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable usage log = true 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: debug monc = 20 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: debug objecter = 20 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: debug rados = 20 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.a] 2024-03-29T11:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.c] 2024-03-29T11:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.b] 2024-03-29T11:34:21.415 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-29T11:34:21.415 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-29T11:34:21.415 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:34:21.415 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-29T11:34:21.415 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.415 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-29T11:34:21.415 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-29T11:34:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "[v1:172.21.15.37:6789],[v1:172.21.15.37:6790],[v1:172.21.15.146:6789]" 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: lockdep = True 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: mon client directed command retry = 5 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: mon election default strategy = 1 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: ms bind msgr2 = False 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: ms inject socket failures = 5000 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: ms type = async 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-29T11:34:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-29T11:34:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-29T11:34:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:34:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-29T11:34:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-29T11:34:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-29T11:34:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-29T11:34:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-29T11:34:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-29T11:34:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-29T11:34:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore compression algorithm = snappy 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore compression mode = aggressive 2024-03-29T11:34:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore zero block detection = True 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 20 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 20 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: debug monc = 20 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 10 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: osd class default list = * 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: osd class load list = * 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: osd client watch timeout = 120 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify cached snaps = True 2024-03-29T11:34:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify missing on start = True 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock override recovery settings = True 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: osd op queue = debug_random 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: osd op queue cut off = debug_random 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: osd shutdown pgref assert = True 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-29T11:34:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:34:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: mon min osdmap epochs = 50 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: mon osdmap full prune interval = 2 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: mon osdmap full prune min = 15 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: mon scrub interval = 300 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on pool no app = False 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: paxos service trim min = 10 2024-03-29T11:34:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-29T11:34:21.425 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-29T11:34:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-29T11:34:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-29T11:34:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-29T11:34:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:34:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:34:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: debug monc = 20 2024-03-29T11:34:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-29T11:34:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: debug objecter = 20 2024-03-29T11:34:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: debug rados = 20 2024-03-29T11:34:21.425 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-29T11:34:21.425 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-29T11:34:21.425 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-29T11:34:21.427 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:34:21.427 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:34:21.482 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:34:21.482 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:21.482 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:34:21.499 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:21.500 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:34:21.554 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi037.front.sepia.ceph.com 2024-03-29T11:34:21.554 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:21.555 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:34:21.555 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:34:21.633 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:21.634 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:21.688 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2024-03-29T11:34:21.688 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:21.688 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:34:21.688 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:34:21.728 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:21.729 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:21.784 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:34:21.784 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:34:21.784 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:34:21.838 INFO:teuthology.orchestra.run.smithi037.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:34:21.841 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:34:21.902 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:34:21.906 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:34:21.907 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:34:21.907 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:34:21.948 INFO:teuthology.orchestra.run.smithi037.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:34:21.963 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:34:22.005 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:34:22.021 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:34:22.021 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:34:22.022 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:22.048 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:22.048 INFO:tasks.ceph:role: osd.0 2024-03-29T11:34:22.048 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-29T11:34:22.048 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:34:22.122 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:22.122 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:22.122 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:22.123 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:22.123 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:22.123 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:22.123 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:22.123 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:22.123 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:22.123 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:22.629 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-03-29T11:34:22.631 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-03-29T11:34:22.631 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:22.747 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:22.781 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:22.847 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:22.848 INFO:tasks.ceph:role: osd.1 2024-03-29T11:34:22.848 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-29T11:34:22.848 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:34:22.921 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:22.921 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:22.921 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:22.921 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:22.922 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:22.922 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:22.922 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:22.922 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:22.922 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:22.922 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:23.369 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-03-29T11:34:23.371 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-03-29T11:34:23.371 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:23.462 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:23.493 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:23.561 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:23.562 INFO:tasks.ceph:role: osd.2 2024-03-29T11:34:23.562 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-29T11:34:23.562 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:34:23.635 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:23.635 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:23.635 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-03-29T11:34:24.128 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-03-29T11:34:24.129 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:24.250 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:24.281 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:24.349 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:24.349 INFO:tasks.ceph:role: osd.3 2024-03-29T11:34:24.349 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-29T11:34:24.349 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:34:24.422 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:24.422 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:24.422 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:24.423 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:24.423 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:24.423 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:24.423 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:24.423 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:24.423 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:24.423 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:24.937 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-03-29T11:34:24.939 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-03-29T11:34:24.940 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:25.053 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:25.086 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:25.188+0000 7f29ea5e1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:25.189+0000 7f29ea5e1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:25.189+0000 7f29ea5e1640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:25.189+0000 7f29ea5e1640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:25.189+0000 7f29ea5e1640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:25.190+0000 7f29ea5e1640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:34:28.038 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:28.068 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:28.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:28.164+0000 7faaacc9b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:34:28.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:28.164+0000 7faaacc9b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:34:28.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:28.164+0000 7faaacc9b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:28.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:28.164+0000 7faaacc9b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:28.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:28.164+0000 7faaacc9b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:28.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:28.165+0000 7faaacc9b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:34:31.010 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:31.041 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:31.135+0000 7f8f63c0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:34:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:31.136+0000 7f8f63c0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:34:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:31.136+0000 7f8f63c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:31.136+0000 7f8f63c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:31.136+0000 7f8f63c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:31.136+0000 7f8f63c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:34:33.983 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:34.014 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:34.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:34.109+0000 7fc3c380c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:34:34.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:34.109+0000 7fc3c380c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:34:34.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:34.109+0000 7fc3c380c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:34.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:34.109+0000 7fc3c380c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:34.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:34.109+0000 7fc3c380c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:34.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:34.110+0000 7fc3c380c640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:34:36.956 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:36.987 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:37.019 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:37.019 INFO:tasks.ceph:role: osd.4 2024-03-29T11:34:37.020 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-29T11:34:37.020 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:34:37.098 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:37.175 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:37.175 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:37.176 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:37.176 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:37.606 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-29T11:34:37.608 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-29T11:34:37.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:37.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:37.756 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:37.824 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:37.824 INFO:tasks.ceph:role: osd.5 2024-03-29T11:34:37.824 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-29T11:34:37.824 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:34:37.910 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:37.910 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:37.910 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:37.910 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:37.910 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:37.910 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:37.910 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:37.910 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:37.911 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:37.911 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:38.364 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-29T11:34:38.367 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-29T11:34:38.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:38.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:38.480 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:38.546 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:38.546 INFO:tasks.ceph:role: osd.6 2024-03-29T11:34:38.547 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-29T11:34:38.547 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:34:38.621 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:38.621 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:38.621 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:38.621 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:38.621 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:38.621 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:38.621 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:38.621 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:38.621 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:38.621 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:39.118 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-29T11:34:39.120 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-29T11:34:39.121 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:39.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:39.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:39.371 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:39.371 INFO:tasks.ceph:role: osd.7 2024-03-29T11:34:39.371 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-29T11:34:39.371 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:34:39.446 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:39.446 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:39.446 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:39.446 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:39.446 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:39.446 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:39.446 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:39.446 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:39.447 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:39.447 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:39.967 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-29T11:34:39.969 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-29T11:34:39.969 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:40.109 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:40.142 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:40.247+0000 7f4909414640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:34:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:40.247+0000 7f4909414640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:34:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:40.247+0000 7f4909414640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:40.247+0000 7f4909414640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:40.247+0000 7f4909414640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:40.248+0000 7f4909414640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:34:43.093 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:43.125 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:43.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:43.220+0000 7fab7dbfa640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:34:43.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:43.220+0000 7fab7dbfa640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:34:43.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:43.220+0000 7fab7dbfa640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:43.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:43.220+0000 7fab7dbfa640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:43.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:43.220+0000 7fab7dbfa640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:43.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:43.221+0000 7fab7dbfa640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:34:46.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:46.098 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:46.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:46.192+0000 7feaf2e11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:34:46.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:46.193+0000 7feaf2e11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:34:46.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:46.193+0000 7feaf2e11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:46.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:46.193+0000 7feaf2e11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:46.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:46.193+0000 7feaf2e11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:46.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:46.193+0000 7feaf2e11640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:34:49.046 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:49.074 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:49.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:49.163+0000 7f85e3719640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:34:49.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:49.163+0000 7f85e3719640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:34:49.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:49.163+0000 7f85e3719640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:49.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:49.163+0000 7f85e3719640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:49.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:49.163+0000 7f85e3719640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:49.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:34:49.164+0000 7f85e3719640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:34:52.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:52.040 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:34:52.040 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:52.040 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:34:52.072 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:52.072 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:34:52.143 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:52.144 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:34:52.214 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:52.214 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:34:52.284 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:52.284 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:34:52.354 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:52.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:34:52.385 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:52.385 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:34:52.455 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:52.455 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:34:52.523 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:52.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:34:52.592 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:52.592 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:34:52.660 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:52.660 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:34:52.678 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:52.678 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:34:52.715 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:34:52.715 DEBUG:teuthology.orchestra.run.smithi037:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:34:52.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:34:52.764 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr.y] 2024-03-29T11:34:52.764 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA9pwZm8OI5MhAAGrl5lXv5MlcoFpWi24toew== 2024-03-29T11:34:52.764 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.0] 2024-03-29T11:34:52.764 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBBpwZmsbVLCxAAXD4UHh84SVHLbMPOAQl0CQ== 2024-03-29T11:34:52.764 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.1] 2024-03-29T11:34:52.764 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBEpwZmkNvUCRAAAY7ZtmoqM2tnwDbz2/DM3g== 2024-03-29T11:34:52.764 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.2] 2024-03-29T11:34:52.764 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBHpwZm1RQjCBAA7AGgJottmAemS8iUCYgbEQ== 2024-03-29T11:34:52.764 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.3] 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBKpwZmSXeTBhAART118CabpDb3IROKrrVRQA== 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr.x] 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA9pwZm1V0MNhAA8ztvI89ye4MnSE2AZ/zBHg== 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.4] 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBQpwZm7KDLDhAAytfHBRMB3KgdpIBwNhDQlA== 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.5] 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBTpwZmwfYtDRAAIhuqJgXlVdX/T96JlfT3RQ== 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.6] 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBWpwZmNuOJCxAA45fmNdeZsF5uiAb52QT3rw== 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.7] 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBZpwZmpc/ECRAAEGMsbWEFlmeVBAuGjD48rg== 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout:[client.0] 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA9pwZmcgPLOBAA1pYnx8F7Fcz5vdy7BGigAg== 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout:[client.1] 2024-03-29T11:34:52.765 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA+pwZmZEeWABAAzTf6QeL5xBo9df9vvFs+pw== 2024-03-29T11:34:52.781 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.y] 2024-03-29T11:34:52.781 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA9pwZm8OI5MhAAGrl5lXv5MlcoFpWi24toew== 2024-03-29T11:34:52.781 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.0] 2024-03-29T11:34:52.781 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBBpwZmsbVLCxAAXD4UHh84SVHLbMPOAQl0CQ== 2024-03-29T11:34:52.781 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.1] 2024-03-29T11:34:52.781 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBEpwZmkNvUCRAAAY7ZtmoqM2tnwDbz2/DM3g== 2024-03-29T11:34:52.781 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.2] 2024-03-29T11:34:52.781 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBHpwZm1RQjCBAA7AGgJottmAemS8iUCYgbEQ== 2024-03-29T11:34:52.782 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.3] 2024-03-29T11:34:52.782 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBKpwZmSXeTBhAART118CabpDb3IROKrrVRQA== 2024-03-29T11:34:52.782 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.x] 2024-03-29T11:34:52.782 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA9pwZm1V0MNhAA8ztvI89ye4MnSE2AZ/zBHg== 2024-03-29T11:34:52.782 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.4] 2024-03-29T11:34:52.782 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBQpwZm7KDLDhAAytfHBRMB3KgdpIBwNhDQlA== 2024-03-29T11:34:52.782 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.5] 2024-03-29T11:34:52.782 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBTpwZmwfYtDRAAIhuqJgXlVdX/T96JlfT3RQ== 2024-03-29T11:34:52.782 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.6] 2024-03-29T11:34:52.782 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBWpwZmNuOJCxAA45fmNdeZsF5uiAb52QT3rw== 2024-03-29T11:34:52.782 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.7] 2024-03-29T11:34:52.782 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBZpwZmpc/ECRAAEGMsbWEFlmeVBAuGjD48rg== 2024-03-29T11:34:52.782 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2024-03-29T11:34:52.782 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA9pwZmcgPLOBAA1pYnx8F7Fcz5vdy7BGigAg== 2024-03-29T11:34:52.783 INFO:teuthology.orchestra.run.smithi146.stdout:[client.1] 2024-03-29T11:34:52.783 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA+pwZmZEeWABAAzTf6QeL5xBo9df9vvFs+pw== 2024-03-29T11:34:52.786 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:52.814 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:52.894 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:52.903 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:52.984 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:52.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.050 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.111 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.169 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:53.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:53.229 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.235 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.288 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.294 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.348 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.407 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.462 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:53.465 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:53.520 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:53.523 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:53.579 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:34:53.579 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:34:53.606 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:53.702 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:34:53.728 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:34:53.794 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:53.891 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:34:53.917 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:34:53.944 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:54.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:34:54.070 DEBUG:teuthology.orchestra.run.smithi037:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:54.072 DEBUG:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:54.126 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:34:54.126 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:34:54.126 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:34:54.128 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:34:54.128 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:34:54.128 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:34:54.130 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:34:54.130 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:34:54.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:34:54.168 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:34:54.168 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:34:54.168 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:34:54.169 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:34:54.171 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:34:54.171 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:34:54.171 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:34:54.173 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:34:54.173 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:34:54.173 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:34:54.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.278+0000 7fdd6c2fb640 10 monclient: get_monmap_and_config 2024-03-29T11:34:54.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.278+0000 7fdd6c2fb640 10 monclient: build_initial_monmap 2024-03-29T11:34:54.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.278+0000 7fdd6c2fb640 10 monclient: monmap: 2024-03-29T11:34:54.276 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-03-29T11:34:54.276 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:34:54.276 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-29T11:34:54.279382+0000 2024-03-29T11:34:54.276 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-29T11:34:54.279382+0000 2024-03-29T11:34:54.276 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:54.277 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:34:54.277 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:34:54.277 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:34:54.277 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:34:54.277 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:34:54.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.279+0000 7fdd6c2fb640 1 Processor -- start 2024-03-29T11:34:54.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.279+0000 7fdd6c2fb640 1 -- start start 2024-03-29T11:34:54.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.279+0000 7fdd6c2fb640 10 monclient: init 2024-03-29T11:34:54.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.279+0000 7fdd6c2fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:54.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.279+0000 7fdd6c2fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:54.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.279+0000 7fdd6c2fb640 10 monclient(hunting): picked mon.noname-c con 0x7fdd6409ac40 addr v1:172.21.15.146:6789/0 2024-03-29T11:34:54.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.279+0000 7fdd6c2fb640 10 monclient(hunting): picked mon.noname-a con 0x7fdd6409c870 addr v1:172.21.15.37:6789/0 2024-03-29T11:34:54.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.279+0000 7fdd6c2fb640 10 monclient(hunting): picked mon.noname-b con 0x7fdd640b4360 addr v1:172.21.15.37:6790/0 2024-03-29T11:34:54.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.279+0000 7fdd6c2fb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6409dec0 con 0x7fdd6409c870 2024-03-29T11:34:54.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.279+0000 7fdd6c2fb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6409da30 con 0x7fdd6409ac40 2024-03-29T11:34:54.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.279+0000 7fdd6c2fb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd640b39c0 con 0x7fdd640b4360 2024-03-29T11:34:54.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.279+0000 7fdd6c2fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:54.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.279+0000 7fdd6c2fb640 10 monclient(hunting): authenticate will time out at 873.728845s 2024-03-29T11:34:54.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.280+0000 7fdd6a898640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fdd640b4360 0x7fdd6414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:43720/0 (socket says 172.21.15.37:43720) 2024-03-29T11:34:54.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:54.280+0000 7fdd6a898640 1 -- 172.21.15.37:0/2128907876 learned_addr learned my addr 172.21.15.37:0/2128907876 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:34:57.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 10 monclient(hunting): tick 2024-03-29T11:34:57.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:34:57.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 1 monclient(hunting): continuing hunt 2024-03-29T11:34:57.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:34:57.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 1 -- 172.21.15.37:0/2128907876 >> v1:172.21.15.37:6790/0 conn(0x7fdd640b4360 legacy=0x7fdd6414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 1 -- 172.21.15.37:0/2128907876 >> v1:172.21.15.146:6789/0 conn(0x7fdd6409ac40 legacy=0x7fdd640b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 1 -- 172.21.15.37:0/2128907876 >> v1:172.21.15.37:6789/0 conn(0x7fdd6409c870 legacy=0x7fdd6413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:57.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 10 monclient(hunting): picked mon.noname-c con 0x7fdd500036e0 addr v1:172.21.15.146:6789/0 2024-03-29T11:34:57.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 10 monclient(hunting): picked mon.noname-b con 0x7fdd500072d0 addr v1:172.21.15.37:6790/0 2024-03-29T11:34:57.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 10 monclient(hunting): picked mon.noname-a con 0x7fdd5000b020 addr v1:172.21.15.37:6789/0 2024-03-29T11:34:57.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 1 -- 172.21.15.37:0/2128907876 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd640b5160 con 0x7fdd5000b020 2024-03-29T11:34:57.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 1 -- 172.21.15.37:0/2128907876 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5000fe10 con 0x7fdd500036e0 2024-03-29T11:34:57.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 1 -- 172.21.15.37:0/2128907876 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd50011050 con 0x7fdd500072d0 2024-03-29T11:34:57.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:57.280+0000 7fdd63fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:59.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.446+0000 7fdd69095640 1 -- 172.21.15.37:0/2128907876 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989881499 0 0) 0x7fdd640b5160 con 0x7fdd5000b020 2024-03-29T11:34:59.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.446+0000 7fdd69095640 10 monclient(hunting): _negotiate 2024-03-29T11:34:59.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.446+0000 7fdd69095640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:59.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.446+0000 7fdd69095640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:59.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.446+0000 7fdd69095640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:34:59.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.446+0000 7fdd69095640 1 -- 172.21.15.37:0/2128907876 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd4c003520 con 0x7fdd5000b020 2024-03-29T11:34:59.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 1 -- 172.21.15.37:0/2128907876 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336652775 0 0) 0x7fdd5000fe10 con 0x7fdd500036e0 2024-03-29T11:34:59.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient(hunting): _negotiate 2024-03-29T11:34:59.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:59.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:59.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:34:59.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 1 -- 172.21.15.37:0/2128907876 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd640b5160 con 0x7fdd500036e0 2024-03-29T11:34:59.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 1 -- 172.21.15.37:0/2128907876 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3242811934 0 0) 0x7fdd640b5160 con 0x7fdd500036e0 2024-03-29T11:34:59.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 1 -- 172.21.15.37:0/2128907876 >> v1:172.21.15.37:6790/0 conn(0x7fdd500072d0 legacy=0x7fdd50009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:59.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 1 -- 172.21.15.37:0/2128907876 >> v1:172.21.15.37:6789/0 conn(0x7fdd5000b020 legacy=0x7fdd5000d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:59.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient: _finish_hunting 0 2024-03-29T11:34:59.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 1 monclient: found mon.noname-c 2024-03-29T11:34:59.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:59.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:34:59.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 1 -- 172.21.15.37:0/2128907876 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd50012290 con 0x7fdd500036e0 2024-03-29T11:34:59.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient: _finish_auth 0 2024-03-29T11:34:59.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient: _check_auth_tickets 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 1 -- 172.21.15.37:0/2128907876 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fdd5c001610 con 0x7fdd500036e0 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd6c2fb640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd6c2fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient: dump: 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:34:59.449 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:34:59.450 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:34:59.450 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:34:59.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd6c2fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:59.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 1 -- 172.21.15.37:0/2128907876 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd5c001770 con 0x7fdd500036e0 2024-03-29T11:34:59.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:59.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 1 -- 172.21.15.37:0/2128907876 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fdd5c001b80 con 0x7fdd500036e0 2024-03-29T11:34:59.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:59.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:59.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd69095640 10 monclient: dump: 2024-03-29T11:34:59.450 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-29T11:34:59.450 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:34:59.450 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:34:59.450 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:34:59.450 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd6c2fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.447+0000 7fdd6c2fb640 4 set_mon_vals no callback set 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.448+0000 7fdd6c2fb640 10 monclient: shutdown 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.448+0000 7fdd6c2fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.448+0000 7fdd6c2fb640 1 -- 172.21.15.37:0/2128907876 >> v1:172.21.15.146:6789/0 conn(0x7fdd500036e0 legacy=0x7fdd50005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.448+0000 7fdd6c2fb640 1 -- 172.21.15.37:0/2128907876 shutdown_connections 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.448+0000 7fdd6c2fb640 1 -- 172.21.15.37:0/2128907876 >> 172.21.15.37:0/2128907876 conn(0x7fdd64066b00 msgr2=0x7fdd64066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.448+0000 7fdd6c2fb640 1 -- 172.21.15.37:0/2128907876 shutdown_connections 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.448+0000 7fdd6c2fb640 1 -- 172.21.15.37:0/2128907876 wait complete. 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.449+0000 7fdd6c2fb640 10 monclient: build_initial_monmap 2024-03-29T11:34:59.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.449+0000 7fdd6c2fb640 10 monclient: monmap: 2024-03-29T11:34:59.452 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-03-29T11:34:59.452 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:59.452 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-03-29T11:34:59.452 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-03-29T11:34:59.452 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:59.452 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:34:59.452 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:34:59.452 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:34:59.452 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:34:59.452 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:34:59.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.450+0000 7fdd6c2fb640 1 librados: starting msgr at 2024-03-29T11:34:59.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.450+0000 7fdd6c2fb640 1 librados: starting objecter 2024-03-29T11:34:59.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.450+0000 7fdd6c2fb640 1 Processor -- start 2024-03-29T11:34:59.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.450+0000 7fdd6c2fb640 1 -- start start 2024-03-29T11:34:59.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.450+0000 7fdd6c2fb640 1 librados: setting wanted keys 2024-03-29T11:34:59.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.450+0000 7fdd6c2fb640 1 librados: calling monclient init 2024-03-29T11:34:59.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.450+0000 7fdd6c2fb640 10 monclient: init 2024-03-29T11:34:59.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd6c2fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:59.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd6c2fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:59.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd6c2fb640 10 monclient(hunting): picked mon.noname-b con 0x7fdd500036e0 addr v1:172.21.15.146:6789/0 2024-03-29T11:34:59.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd6c2fb640 10 monclient(hunting): picked mon.noname-a con 0x7fdd500072d0 addr v1:172.21.15.37:6789/0 2024-03-29T11:34:59.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd6c2fb640 10 monclient(hunting): picked mon.noname-c con 0x7fdd5000b020 addr v1:172.21.15.37:6790/0 2024-03-29T11:34:59.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd6c2fb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd64141040 con 0x7fdd500072d0 2024-03-29T11:34:59.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd6c2fb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd64137830 con 0x7fdd500036e0 2024-03-29T11:34:59.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd6c2fb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd640b8110 con 0x7fdd5000b020 2024-03-29T11:34:59.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd6c2fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:59.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd6c2fb640 10 monclient(hunting): authenticate will time out at 878.901350s 2024-03-29T11:34:59.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd6a097640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fdd500036e0 0x7fdd641da9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:43952/0 (socket says 172.21.15.37:43952) 2024-03-29T11:34:59.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd6a097640 1 -- 172.21.15.37:0/647912749 learned_addr learned my addr 172.21.15.37:0/647912749 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:34:59.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd69896640 1 --1- 172.21.15.37:0/647912749 >> v1:172.21.15.37:6789/0 conn(0x7fdd500072d0 0x7fdd64136a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41070/0 (socket says 172.21.15.37:41070) 2024-03-29T11:34:59.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd62ffd640 10 client.?.objecter ms_handle_connect 0x7fdd500036e0 2024-03-29T11:34:59.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:59.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd62ffd640 10 client.?.objecter ms_handle_connect 0x7fdd5000b020 2024-03-29T11:34:59.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:59.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd62ffd640 10 client.?.objecter ms_handle_connect 0x7fdd500072d0 2024-03-29T11:34:59.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:59.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592543178 0 0) 0x7fdd64137830 con 0x7fdd500036e0 2024-03-29T11:34:59.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd62ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:59.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:59.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:59.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd62ffd640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:34:59.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.451+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd500119a0 con 0x7fdd500036e0 2024-03-29T11:34:59.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147347294 0 0) 0x7fdd640b8110 con 0x7fdd5000b020 2024-03-29T11:34:59.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:59.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:59.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:59.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:34:59.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd64137830 con 0x7fdd5000b020 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 171864190 0 0) 0x7fdd500119a0 con 0x7fdd500036e0 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd640b8110 con 0x7fdd500036e0 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fdd5c001d80 con 0x7fdd500036e0 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:34:59.460 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 857014141 0 0) 0x7fdd64137830 con 0x7fdd5000b020 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd500119a0 con 0x7fdd5000b020 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fdd54001c30 con 0x7fdd5000b020 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:34:59.461 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:34:59.462 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:34:59.462 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:34:59.462 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:34:59.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345293423 0 0) 0x7fdd64141040 con 0x7fdd500072d0 2024-03-29T11:34:59.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:59.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:59.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:59.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:34:59.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.452+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd64137830 con 0x7fdd500072d0 2024-03-29T11:34:59.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.453+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3929660088 0 0) 0x7fdd64137830 con 0x7fdd500072d0 2024-03-29T11:34:59.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.453+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd64141040 con 0x7fdd500072d0 2024-03-29T11:34:59.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.453+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fdd58001610 con 0x7fdd500072d0 2024-03-29T11:34:59.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.453+0000 7fdd62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.453+0000 7fdd62ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.453+0000 7fdd62ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3865488269 0 0) 0x7fdd500119a0 con 0x7fdd5000b020 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 >> v1:172.21.15.146:6789/0 conn(0x7fdd500036e0 legacy=0x7fdd641da9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:59.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 >> v1:172.21.15.37:6789/0 conn(0x7fdd500072d0 legacy=0x7fdd64136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd62ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd62ffd640 1 monclient: found mon.c 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd62ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd640b83f0 con 0x7fdd5000b020 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd62ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd62ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd6c2fb640 5 monclient: authenticate success, global_id 4106 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd6c2fb640 10 monclient: _renew_subs 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd6c2fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd6c2fb640 1 -- 172.21.15.37:0/647912749 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd640b6bc0 con 0x7fdd5000b020 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd6c2fb640 10 client.4106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd6c2fb640 10 monclient: _renew_subs 2024-03-29T11:34:59.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd6c2fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd6c2fb640 1 -- 172.21.15.37:0/647912749 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd640b6170 con 0x7fdd5000b020 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.454+0000 7fdd6c2fb640 1 librados: init done 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.455+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd54004b60 con 0x7fdd5000b020 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.455+0000 7fdd62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.455+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fdd54002100 con 0x7fdd5000b020 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.455+0000 7fdd62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.455+0000 7fdd62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.455+0000 7fdd68894640 4 set_mon_vals no callback set 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.455+0000 7fdd62ffd640 10 monclient: dump: 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:59.465 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:34:59.466 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:34:59.466 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:34:59.466 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:34:59.466 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:34:59.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.455+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdd54005200 con 0x7fdd5000b020 2024-03-29T11:34:59.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.455+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 930301273 0 0) 0x7fdd54006780 con 0x7fdd5000b020 2024-03-29T11:34:59.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.455+0000 7fdd62ffd640 10 client.4106.objecter ms_dispatch 0x7fdd64000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:34:59.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.455+0000 7fdd62ffd640 3 client.4106.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:34:59.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.455+0000 7fdd62ffd640 3 client.4106.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:34:59.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.455+0000 7fdd62ffd640 20 client.4106.objecter dump_active .. 0 homeless 2024-03-29T11:34:59.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.456+0000 7fdd6c2fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:59.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.456+0000 7fdd6c2fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:59.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.456+0000 7fdd6c2fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:34:59.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.456+0000 7fdd6c2fb640 1 -- 172.21.15.37:0/647912749 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd44005080 con 0x7fdd5000b020 2024-03-29T11:34:59.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.458+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdd540069b0 con 0x7fdd5000b020 2024-03-29T11:34:59.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.458+0000 7fdd62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:59.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.458+0000 7fdd62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:59.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.530+0000 7fdd6c2fb640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:34:59.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.530+0000 7fdd6c2fb640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:34:59.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.530+0000 7fdd6c2fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:34:59.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.530+0000 7fdd6c2fb640 1 -- 172.21.15.37:0/647912749 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fdd44005330 con 0x7fdd5000b020 2024-03-29T11:34:59.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.535+0000 7fdd62ffd640 1 -- 172.21.15.37:0/647912749 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fdd54005810 con 0x7fdd5000b020 2024-03-29T11:34:59.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.535+0000 7fdd62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:34:59.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.535+0000 7fdd62ffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:34:59.534 INFO:teuthology.orchestra.run.smithi037.stderr:adjusted tunables profile to default 2024-03-29T11:34:59.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.536+0000 7fdd6c2fb640 10 librados: watch_flush enter 2024-03-29T11:34:59.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.536+0000 7fdd6c2fb640 10 librados: watch_flush exit 2024-03-29T11:34:59.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.536+0000 7fdd6c2fb640 20 client.4106.objecter shutdown clearing up homeless session... 2024-03-29T11:34:59.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.536+0000 7fdd6c2fb640 10 client.4106.objecter successfully canceled tick 2024-03-29T11:34:59.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.536+0000 7fdd6c2fb640 10 monclient: shutdown 2024-03-29T11:34:59.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.536+0000 7fdd6c2fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:59.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.536+0000 7fdd6c2fb640 1 -- 172.21.15.37:0/647912749 >> v1:172.21.15.37:6790/0 conn(0x7fdd5000b020 legacy=0x7fdd64137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:59.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.536+0000 7fdd6c2fb640 1 -- 172.21.15.37:0/647912749 shutdown_connections 2024-03-29T11:34:59.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.536+0000 7fdd6c2fb640 1 -- 172.21.15.37:0/647912749 >> 172.21.15.37:0/647912749 conn(0x7fdd64066b00 msgr2=0x7fdd6413d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:59.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.536+0000 7fdd6c2fb640 1 -- 172.21.15.37:0/647912749 shutdown_connections 2024-03-29T11:34:59.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.536+0000 7fdd6c2fb640 1 -- 172.21.15.37:0/647912749 wait complete. 2024-03-29T11:34:59.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:34:59.536+0000 7fdd6c2fb640 1 librados: shutdown 2024-03-29T11:34:59.546 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:34:59.546 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:34:59.546 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:59.546 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:34:59.573 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:59.573 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:34:59.640 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:59.640 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:34:59.706 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-29T11:34:59.706 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:34:59.772 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:59.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:34:59.800 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:59.800 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:34:59.868 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:59.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:34:59.933 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:34:59.936+0000 7f04d6813040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:34:59.935 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-29T11:34:59.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:34:59.947 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:34:59.949+0000 7f935f24b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:34:59.990 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:34:59.993+0000 7f04d6813040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:35:00.002 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 3c8cbb56-9953-4797-b700-981d68eda10b 0 2024-03-29T11:35:00.003 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:00.004+0000 7f935f24b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:35:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.145+0000 7f9d07bc6640 10 monclient: get_monmap_and_config 2024-03-29T11:35:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.145+0000 7f9d07bc6640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.145+0000 7f9d07bc6640 10 monclient: monmap: 2024-03-29T11:35:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:35:00.146264+0000 2024-03-29T11:35:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:35:00.146264+0000 2024-03-29T11:35:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:00.143 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.146+0000 7f9d07bc6640 1 Processor -- start 2024-03-29T11:35:00.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.146+0000 7f9d07bc6640 1 -- start start 2024-03-29T11:35:00.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.146+0000 7f9d07bc6640 10 monclient: init 2024-03-29T11:35:00.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.146+0000 7f9d07bc6640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.146+0000 7f9d07bc6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:00.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.146+0000 7f9d07bc6640 10 monclient(hunting): picked mon.noname-a con 0x7f9d0009ac40 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:00.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.146+0000 7f9d07bc6640 10 monclient(hunting): picked mon.noname-c con 0x7f9d0009c870 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:00.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.146+0000 7f9d07bc6640 10 monclient(hunting): picked mon.noname-b con 0x7f9d000b4360 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:00.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.146+0000 7f9d07bc6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0009dec0 con 0x7f9d0009ac40 2024-03-29T11:35:00.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.146+0000 7f9d07bc6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0009da30 con 0x7f9d0009c870 2024-03-29T11:35:00.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.146+0000 7f9d07bc6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d000b39c0 con 0x7f9d000b4360 2024-03-29T11:35:00.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d07bc6640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d07bc6640 10 monclient(hunting): authenticate will time out at 866.390882s 2024-03-29T11:35:00.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d05161640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9d0009c870 0x7f9d0013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38980/0 (socket says 172.21.15.146:38980) 2024-03-29T11:35:00.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d05161640 1 -- 172.21.15.146:0/81332827 learned_addr learned my addr 172.21.15.146:0/81332827 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:00.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 1 -- 172.21.15.146:0/81332827 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822692656 0 0) 0x7f9d0009da30 con 0x7f9d0009c870 2024-03-29T11:35:00.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:35:00.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 1 -- 172.21.15.146:0/81332827 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cec003520 con 0x7f9d0009c870 2024-03-29T11:35:00.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 1 -- 172.21.15.146:0/81332827 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100213137 0 0) 0x7f9cf0001110 con 0x7f9d0009ac40 2024-03-29T11:35:00.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:35:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 1 -- 172.21.15.146:0/81332827 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d0009da30 con 0x7f9d0009ac40 2024-03-29T11:35:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 1 -- 172.21.15.146:0/81332827 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921163812 0 0) 0x7f9d000b39c0 con 0x7f9d000b4360 2024-03-29T11:35:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:35:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 1 -- 172.21.15.146:0/81332827 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cf0001110 con 0x7f9d000b4360 2024-03-29T11:35:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 1 -- 172.21.15.146:0/81332827 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3167019544 0 0) 0x7f9cec003520 con 0x7f9d0009c870 2024-03-29T11:35:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 1 -- 172.21.15.146:0/81332827 >> v1:172.21.15.37:6790/0 conn(0x7f9d000b4360 legacy=0x7f9d0014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 1 -- 172.21.15.146:0/81332827 >> v1:172.21.15.37:6789/0 conn(0x7f9d0009ac40 legacy=0x7f9d000b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 1 monclient: found mon.noname-c 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 1 -- 172.21.15.146:0/81332827 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d000b5160 con 0x7f9d0009c870 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 1 -- 172.21.15.146:0/81332827 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9cfc001690 con 0x7f9d0009c870 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d07bc6640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d07bc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient: dump: 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d07bc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 1 -- 172.21.15.146:0/81332827 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cfc001870 con 0x7f9d0009c870 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d07bc6640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 1 -- 172.21.15.146:0/81332827 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9cfc001c80 con 0x7f9d0009c870 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d07bc6640 4 set_mon_vals no callback set 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.147+0000 7f9d04960640 10 monclient: dump: 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.148+0000 7f9d07bc6640 10 monclient: shutdown 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.148+0000 7f9d07bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.148+0000 7f9d07bc6640 1 -- 172.21.15.146:0/81332827 >> v1:172.21.15.146:6789/0 conn(0x7f9d0009c870 legacy=0x7f9d0013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.148+0000 7f9d07bc6640 1 -- 172.21.15.146:0/81332827 shutdown_connections 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.148+0000 7f9d07bc6640 1 -- 172.21.15.146:0/81332827 >> 172.21.15.146:0/81332827 conn(0x7f9d00066b00 msgr2=0x7f9d00066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.148+0000 7f9d07bc6640 1 -- 172.21.15.146:0/81332827 shutdown_connections 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.148+0000 7f9d07bc6640 1 -- 172.21.15.146:0/81332827 wait complete. 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.148+0000 7f9d07bc6640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.148+0000 7f9d07bc6640 10 monclient: monmap: 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:00.159 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 1 librados: starting msgr at 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 1 librados: starting objecter 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 1 Processor -- start 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 1 -- start start 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 1 librados: setting wanted keys 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 1 librados: calling monclient init 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 10 monclient: init 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 10 monclient(hunting): picked mon.noname-a con 0x7f9d0009ac40 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 10 monclient(hunting): picked mon.noname-c con 0x7f9d000acf70 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 10 monclient(hunting): picked mon.noname-b con 0x7f9d001e0100 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d001d7800 con 0x7f9d0009ac40 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d000ade40 con 0x7f9d001e0100 2024-03-29T11:35:00.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d000ae120 con 0x7f9d000acf70 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.149+0000 7f9d07bc6640 10 monclient(hunting): authenticate will time out at 866.393812s 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9d06163640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9d001e0100 0x7f9d001e24f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38994/0 (socket says 172.21.15.146:38994) 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9d06163640 1 -- 172.21.15.146:0/40683339 learned_addr learned my addr 172.21.15.146:0/40683339 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9d05962640 1 --1- 172.21.15.146:0/40683339 >> v1:172.21.15.37:6789/0 conn(0x7f9d0009ac40 0x7f9d00142760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.146:42466/0 (socket says 172.21.15.146:42466) 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 client.?.objecter ms_handle_connect 0x7f9d001e0100 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 client.?.objecter ms_handle_connect 0x7f9d0009ac40 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 client.?.objecter ms_handle_connect 0x7f9d000acf70 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698180545 0 0) 0x7f9d000ade40 con 0x7f9d001e0100 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ce4004eb0 con 0x7f9d001e0100 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678169430 0 0) 0x7f9d001d7800 con 0x7f9d0009ac40 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d000ade40 con 0x7f9d0009ac40 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689452149 0 0) 0x7f9d000ae120 con 0x7f9d000acf70 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d001d7800 con 0x7f9d000acf70 2024-03-29T11:35:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1216836680 0 0) 0x7f9d000ade40 con 0x7f9d0009ac40 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d000ae120 con 0x7f9d0009ac40 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9cf00019e0 con 0x7f9d0009ac40 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): dump: 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.163 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2982731399 0 0) 0x7f9ce4004eb0 con 0x7f9d001e0100 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d000ade40 con 0x7f9d001e0100 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9cf8001b60 con 0x7f9d001e0100 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): dump: 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2311670423 0 0) 0x7f9d001d7800 con 0x7f9d000acf70 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ce4004eb0 con 0x7f9d000acf70 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9cfc001b80 con 0x7f9d000acf70 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient(hunting): dump: 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.165 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.166 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 937405962 0 0) 0x7f9d000ae120 con 0x7f9d0009ac40 2024-03-29T11:35:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 >> v1:172.21.15.37:6790/0 conn(0x7f9d000acf70 legacy=0x7f9d001d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 >> v1:172.21.15.146:6789/0 conn(0x7f9d001e0100 legacy=0x7f9d001e24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 monclient: found mon.a 2024-03-29T11:35:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0013ffa0 con 0x7f9d0009ac40 2024-03-29T11:35:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9d07bc6640 5 monclient: authenticate success, global_id 4116 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9d07bc6640 10 monclient: _renew_subs 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9d07bc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9d07bc6640 1 -- 172.21.15.146:0/40683339 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d001d83a0 con 0x7f9d0009ac40 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9d07bc6640 10 client.4116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9d07bc6640 10 monclient: _renew_subs 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9d07bc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9d07bc6640 1 -- 172.21.15.146:0/40683339 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d001e3df0 con 0x7f9d0009ac40 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9d07bc6640 1 librados: init done 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cf0003610 con 0x7f9d0009ac40 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9cf0003a20 con 0x7f9d0009ac40 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf7fff640 4 set_mon_vals no callback set 2024-03-29T11:35:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 10 monclient: dump: 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9cf0003fd0 con 0x7f9d0009ac40 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.150+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1456931535 0 0) 0x7f9cf00065e0 con 0x7f9d0009ac40 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.151+0000 7f9cf67fc640 10 client.4116.objecter ms_dispatch 0x7f9d00000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.151+0000 7f9cf67fc640 3 client.4116.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.151+0000 7f9cf67fc640 3 client.4116.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:35:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.151+0000 7f9cf67fc640 20 client.4116.objecter dump_active .. 0 homeless 2024-03-29T11:35:00.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.151+0000 7f9d07bc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.151+0000 7f9d07bc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.151+0000 7f9d07bc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:00.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.151+0000 7f9d07bc6640 1 -- 172.21.15.146:0/40683339 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce8005080 con 0x7f9d0009ac40 2024-03-29T11:35:00.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.154+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9cf0004410 con 0x7f9d0009ac40 2024-03-29T11:35:00.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.154+0000 7f9cf67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.154+0000 7f9cf67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:00.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.226+0000 7f9d07bc6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3c8cbb56-9953-4797-b700-981d68eda10b", "id": 0}] 2024-03-29T11:35:00.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.226+0000 7f9d07bc6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3c8cbb56-9953-4797-b700-981d68eda10b", "id": 0}] 2024-03-29T11:35:00.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.226+0000 7f9d07bc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:00.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.226+0000 7f9d07bc6640 1 -- 172.21.15.146:0/40683339 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3c8cbb56-9953-4797-b700-981d68eda10b", "id": 0} v 0) v1 -- 0x7f9ce80058b0 con 0x7f9d0009ac40 2024-03-29T11:35:00.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.228+0000 7f9cf67fc640 1 -- 172.21.15.146:0/40683339 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3c8cbb56-9953-4797-b700-981d68eda10b", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1964723979 0 732707515) 0x7f9cf0005890 con 0x7f9d0009ac40 2024-03-29T11:35:00.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.228+0000 7f9cf67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3c8cbb56-9953-4797-b700-981d68eda10b", "id": 0}] 2024-03-29T11:35:00.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.228+0000 7f9cf67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:00.225 INFO:teuthology.orchestra.run.smithi146.stdout:0 2024-03-29T11:35:00.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.228+0000 7f9d07bc6640 10 librados: watch_flush enter 2024-03-29T11:35:00.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.228+0000 7f9d07bc6640 10 librados: watch_flush exit 2024-03-29T11:35:00.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.228+0000 7f9d07bc6640 20 client.4116.objecter shutdown clearing up homeless session... 2024-03-29T11:35:00.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.228+0000 7f9d07bc6640 10 client.4116.objecter successfully canceled tick 2024-03-29T11:35:00.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.228+0000 7f9d07bc6640 10 monclient: shutdown 2024-03-29T11:35:00.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.228+0000 7f9d07bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.228+0000 7f9d07bc6640 1 -- 172.21.15.146:0/40683339 >> v1:172.21.15.37:6789/0 conn(0x7f9d0009ac40 legacy=0x7f9d00142760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.228+0000 7f9d07bc6640 1 -- 172.21.15.146:0/40683339 shutdown_connections 2024-03-29T11:35:00.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.228+0000 7f9d07bc6640 1 -- 172.21.15.146:0/40683339 >> 172.21.15.146:0/40683339 conn(0x7f9d00066b00 msgr2=0x7f9d0013e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.229+0000 7f9d07bc6640 1 -- 172.21.15.146:0/40683339 shutdown_connections 2024-03-29T11:35:00.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.229+0000 7f9d07bc6640 1 -- 172.21.15.146:0/40683339 wait complete. 2024-03-29T11:35:00.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.229+0000 7f9d07bc6640 1 librados: shutdown 2024-03-29T11:35:00.236 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 9a7adcae-7e22-4730-9149-d05cd77f3034 1 2024-03-29T11:35:00.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.340+0000 7f6a7877b640 10 monclient: get_monmap_and_config 2024-03-29T11:35:00.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.340+0000 7f6a7877b640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.340+0000 7f6a7877b640 10 monclient: monmap: 2024-03-29T11:35:00.338 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:00.338 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.338 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:35:00.340811+0000 2024-03-29T11:35:00.338 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:35:00.340811+0000 2024-03-29T11:35:00.338 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.338 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.338 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:00.338 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:00.338 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:00.338 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.340+0000 7f6a7877b640 1 Processor -- start 2024-03-29T11:35:00.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.340+0000 7f6a7877b640 1 -- start start 2024-03-29T11:35:00.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.340+0000 7f6a7877b640 10 monclient: init 2024-03-29T11:35:00.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a7877b640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a7877b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a7877b640 10 monclient(hunting): picked mon.noname-b con 0x7f6a7009ac40 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a7877b640 10 monclient(hunting): picked mon.noname-c con 0x7f6a7009c870 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a7877b640 10 monclient(hunting): picked mon.noname-a con 0x7f6a700b4360 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a7877b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a7009dec0 con 0x7f6a700b4360 2024-03-29T11:35:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a7877b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a7009da30 con 0x7f6a7009c870 2024-03-29T11:35:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a7877b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a700b39c0 con 0x7f6a7009ac40 2024-03-29T11:35:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a7877b640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a7877b640 10 monclient(hunting): authenticate will time out at 866.585380s 2024-03-29T11:35:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a75d16640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6a7009c870 0x7f6a7013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:39006/0 (socket says 172.21.15.146:39006) 2024-03-29T11:35:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a75d16640 1 -- 172.21.15.146:0/1297799212 learned_addr learned my addr 172.21.15.146:0/1297799212 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a75515640 1 -- 172.21.15.146:0/1297799212 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649986493 0 0) 0x7f6a7009da30 con 0x7f6a7009c870 2024-03-29T11:35:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a75515640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a75515640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a75515640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a75515640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.341+0000 7f6a75515640 1 -- 172.21.15.146:0/1297799212 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a5c003520 con 0x7f6a7009c870 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 1 -- 172.21.15.146:0/1297799212 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965430463 0 0) 0x7f6a68001490 con 0x7f6a700b4360 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 1 -- 172.21.15.146:0/1297799212 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a7009da30 con 0x7f6a700b4360 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 1 -- 172.21.15.146:0/1297799212 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800474053 0 0) 0x7f6a700b39c0 con 0x7f6a7009ac40 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:35:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 1 -- 172.21.15.146:0/1297799212 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a68001490 con 0x7f6a7009ac40 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 1 -- 172.21.15.146:0/1297799212 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3723290303 0 0) 0x7f6a5c003520 con 0x7f6a7009c870 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 1 -- 172.21.15.146:0/1297799212 >> v1:172.21.15.37:6790/0 conn(0x7f6a7009ac40 legacy=0x7f6a700b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 1 -- 172.21.15.146:0/1297799212 >> v1:172.21.15.37:6789/0 conn(0x7f6a700b4360 legacy=0x7f6a7014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 1 monclient: found mon.noname-c 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 1 -- 172.21.15.146:0/1297799212 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a700b5160 con 0x7f6a7009c870 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 1 -- 172.21.15.146:0/1297799212 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6a6c001690 con 0x7f6a7009c870 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a7877b640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a7877b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient: dump: 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a7877b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 1 -- 172.21.15.146:0/1297799212 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a6c001870 con 0x7f6a7009c870 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a7877b640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:00.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 1 -- 172.21.15.146:0/1297799212 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6a6c001c80 con 0x7f6a7009c870 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a7877b640 4 set_mon_vals no callback set 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a75515640 10 monclient: dump: 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a7877b640 10 monclient: shutdown 2024-03-29T11:35:00.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a7877b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a7877b640 1 -- 172.21.15.146:0/1297799212 >> v1:172.21.15.146:6789/0 conn(0x7f6a7009c870 legacy=0x7f6a7013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a7877b640 1 -- 172.21.15.146:0/1297799212 shutdown_connections 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a7877b640 1 -- 172.21.15.146:0/1297799212 >> 172.21.15.146:0/1297799212 conn(0x7f6a70066b00 msgr2=0x7f6a70066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.342+0000 7f6a7877b640 1 -- 172.21.15.146:0/1297799212 shutdown_connections 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.343+0000 7f6a7877b640 1 -- 172.21.15.146:0/1297799212 wait complete. 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.343+0000 7f6a7877b640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.343+0000 7f6a7877b640 10 monclient: monmap: 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.343+0000 7f6a7877b640 1 librados: starting msgr at 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.343+0000 7f6a7877b640 1 librados: starting objecter 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.343+0000 7f6a7877b640 1 Processor -- start 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a7877b640 1 -- start start 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a7877b640 1 librados: setting wanted keys 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a7877b640 1 librados: calling monclient init 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a7877b640 10 monclient: init 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a7877b640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a7877b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a7877b640 10 monclient(hunting): picked mon.noname-a con 0x7f6a7009ac40 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a7877b640 10 monclient(hunting): picked mon.noname-b con 0x7f6a700b6b00 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a7877b640 10 monclient(hunting): picked mon.noname-c con 0x7f6a701406b0 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:00.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a7877b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a701d7e20 con 0x7f6a7009ac40 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a7877b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a70140b00 con 0x7f6a700b6b00 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a7877b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a700ad080 con 0x7f6a701406b0 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a7877b640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a7877b640 10 monclient(hunting): authenticate will time out at 866.588452s 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a75d16640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6a700b6b00 0x7f6a7013ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:39020/0 (socket says 172.21.15.146:39020) 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a75d16640 1 -- 172.21.15.146:0/2087600884 learned_addr learned my addr 172.21.15.146:0/2087600884 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 client.?.objecter ms_handle_connect 0x7f6a700b6b00 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 client.?.objecter ms_handle_connect 0x7f6a7009ac40 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 client.?.objecter ms_handle_connect 0x7f6a701406b0 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358974050 0 0) 0x7f6a70140b00 con 0x7f6a700b6b00 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a54004eb0 con 0x7f6a700b6b00 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070611913 0 0) 0x7f6a701d7e20 con 0x7f6a7009ac40 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a70140b00 con 0x7f6a7009ac40 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668240980 0 0) 0x7f6a700ad080 con 0x7f6a701406b0 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.344+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a701d7e20 con 0x7f6a701406b0 2024-03-29T11:35:00.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 194025695 0 0) 0x7f6a54004eb0 con 0x7f6a700b6b00 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a700ad080 con 0x7f6a700b6b00 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6a6c001bc0 con 0x7f6a700b6b00 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2686194762 0 0) 0x7f6a70140b00 con 0x7f6a7009ac40 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a54004eb0 con 0x7f6a7009ac40 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3313081814 0 0) 0x7f6a701d7e20 con 0x7f6a701406b0 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a70140b00 con 0x7f6a701406b0 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6a600017a0 con 0x7f6a7009ac40 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6a68001a00 con 0x7f6a701406b0 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3063690771 0 0) 0x7f6a700ad080 con 0x7f6a700b6b00 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 >> v1:172.21.15.37:6790/0 conn(0x7f6a701406b0 legacy=0x7f6a701e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 >> v1:172.21.15.37:6789/0 conn(0x7f6a7009ac40 legacy=0x7f6a701d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 monclient: found mon.b 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a701d7ab0 con 0x7f6a700b6b00 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a7877b640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a7877b640 10 monclient: _renew_subs 2024-03-29T11:35:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a7877b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a7877b640 1 -- 172.21.15.146:0/2087600884 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a701e3570 con 0x7f6a700b6b00 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a7877b640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a7877b640 10 monclient: _renew_subs 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a7877b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a7877b640 1 -- 172.21.15.146:0/2087600884 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a701e38e0 con 0x7f6a700b6b00 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a7877b640 1 librados: init done 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a6c004bf0 con 0x7f6a700b6b00 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6a6c004f00 con 0x7f6a700b6b00 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a74d14640 4 set_mon_vals no callback set 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 monclient: dump: 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6a6c005470 con 0x7f6a700b6b00 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3104890335 0 0) 0x7f6a6c0068a0 con 0x7f6a700b6b00 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 10 client.4123.objecter ms_dispatch 0x7f6a70000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 3 client.4123.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.345+0000 7f6a66ffd640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.346+0000 7f6a7877b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.346+0000 7f6a7877b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.346+0000 7f6a7877b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.346+0000 7f6a7877b640 1 -- 172.21.15.146:0/2087600884 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a7009b020 con 0x7f6a700b6b00 2024-03-29T11:35:00.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.348+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6a6c006a60 con 0x7f6a700b6b00 2024-03-29T11:35:00.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.348+0000 7f6a66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.348+0000 7f6a66ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:00.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.420+0000 7f6a7877b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9a7adcae-7e22-4730-9149-d05cd77f3034", "id": 1}] 2024-03-29T11:35:00.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.420+0000 7f6a7877b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9a7adcae-7e22-4730-9149-d05cd77f3034", "id": 1}] 2024-03-29T11:35:00.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.420+0000 7f6a7877b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.420+0000 7f6a7877b640 1 -- 172.21.15.146:0/2087600884 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9a7adcae-7e22-4730-9149-d05cd77f3034", "id": 1} v 0) v1 -- 0x7f6a7004f730 con 0x7f6a700b6b00 2024-03-29T11:35:00.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.428+0000 7f6a66ffd640 1 -- 172.21.15.146:0/2087600884 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9a7adcae-7e22-4730-9149-d05cd77f3034", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2489294644 0 940483276) 0x7f6a6c0058e0 con 0x7f6a700b6b00 2024-03-29T11:35:00.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.428+0000 7f6a66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9a7adcae-7e22-4730-9149-d05cd77f3034", "id": 1}] 2024-03-29T11:35:00.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.428+0000 7f6a66ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:00.426 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-03-29T11:35:00.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.429+0000 7f6a7877b640 10 librados: watch_flush enter 2024-03-29T11:35:00.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.429+0000 7f6a7877b640 10 librados: watch_flush exit 2024-03-29T11:35:00.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.429+0000 7f6a7877b640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T11:35:00.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.429+0000 7f6a7877b640 10 client.4123.objecter successfully canceled tick 2024-03-29T11:35:00.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.429+0000 7f6a7877b640 10 monclient: shutdown 2024-03-29T11:35:00.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.429+0000 7f6a7877b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.429+0000 7f6a7877b640 1 -- 172.21.15.146:0/2087600884 >> v1:172.21.15.146:6789/0 conn(0x7f6a700b6b00 legacy=0x7f6a7013ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.429+0000 7f6a7877b640 1 -- 172.21.15.146:0/2087600884 shutdown_connections 2024-03-29T11:35:00.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.429+0000 7f6a7877b640 1 -- 172.21.15.146:0/2087600884 >> 172.21.15.146:0/2087600884 conn(0x7f6a70066b00 msgr2=0x7f6a700b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.429+0000 7f6a7877b640 1 -- 172.21.15.146:0/2087600884 shutdown_connections 2024-03-29T11:35:00.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.429+0000 7f6a7877b640 1 -- 172.21.15.146:0/2087600884 wait complete. 2024-03-29T11:35:00.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.429+0000 7f6a7877b640 1 librados: shutdown 2024-03-29T11:35:00.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 9d447705-705c-4657-ba3d-5ab095b713a1 2 2024-03-29T11:35:00.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.539+0000 7f777b07b640 10 monclient: get_monmap_and_config 2024-03-29T11:35:00.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.539+0000 7f777b07b640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.540+0000 7f777b07b640 10 monclient: monmap: 2024-03-29T11:35:00.537 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:00.537 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.538 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:35:00.540682+0000 2024-03-29T11:35:00.538 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:35:00.540682+0000 2024-03-29T11:35:00.538 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.538 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.538 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:00.538 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:00.538 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:00.538 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.540+0000 7f777b07b640 1 Processor -- start 2024-03-29T11:35:00.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.540+0000 7f777b07b640 1 -- start start 2024-03-29T11:35:00.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.540+0000 7f777b07b640 10 monclient: init 2024-03-29T11:35:00.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f777b07b640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f777b07b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:00.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f777b07b640 10 monclient(hunting): picked mon.noname-a con 0x7f777409ac40 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:00.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f777b07b640 10 monclient(hunting): picked mon.noname-b con 0x7f777409c870 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:00.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f777b07b640 10 monclient(hunting): picked mon.noname-c con 0x7f77740b4360 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f777b07b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777409dec0 con 0x7f777409ac40 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f777b07b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777409da30 con 0x7f77740b4360 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f777b07b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77740b39c0 con 0x7f777409c870 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f777b07b640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f777b07b640 10 monclient(hunting): authenticate will time out at 866.785238s 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f7779618640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f77740b4360 0x7f777414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:39036/0 (socket says 172.21.15.146:39036) 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f7779618640 1 -- 172.21.15.146:0/416719434 learned_addr learned my addr 172.21.15.146:0/416719434 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 1 -- 172.21.15.146:0/416719434 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090771792 0 0) 0x7f777409da30 con 0x7f77740b4360 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 1 -- 172.21.15.146:0/416719434 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7764003520 con 0x7f77740b4360 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 1 -- 172.21.15.146:0/416719434 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040720481 0 0) 0x7f7768001110 con 0x7f777409ac40 2024-03-29T11:35:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 1 -- 172.21.15.146:0/416719434 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f777409da30 con 0x7f777409ac40 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 1 -- 172.21.15.146:0/416719434 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556588934 0 0) 0x7f77740b39c0 con 0x7f777409c870 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 1 -- 172.21.15.146:0/416719434 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7768001110 con 0x7f777409c870 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 1 -- 172.21.15.146:0/416719434 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3170640345 0 0) 0x7f7764003520 con 0x7f77740b4360 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 1 -- 172.21.15.146:0/416719434 >> v1:172.21.15.37:6790/0 conn(0x7f777409c870 legacy=0x7f777413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 1 -- 172.21.15.146:0/416719434 >> v1:172.21.15.37:6789/0 conn(0x7f777409ac40 legacy=0x7f77740b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 1 monclient: found mon.noname-c 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.541+0000 7f77737fe640 1 -- 172.21.15.146:0/416719434 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77740b5160 con 0x7f77740b4360 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f77737fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f77737fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f77737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f77737fe640 1 -- 172.21.15.146:0/416719434 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f776c001a10 con 0x7f77740b4360 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f777b07b640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f777b07b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f77737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f77737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f77737fe640 10 monclient: dump: 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.545 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f777b07b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f77737fe640 1 -- 172.21.15.146:0/416719434 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f776c001bf0 con 0x7f77740b4360 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f77737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f77737fe640 1 -- 172.21.15.146:0/416719434 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f776c002000 con 0x7f77740b4360 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f777b07b640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f777b07b640 4 set_mon_vals no callback set 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f77737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f77737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f77737fe640 10 monclient: dump: 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f777b07b640 10 monclient: shutdown 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f777b07b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f777b07b640 1 -- 172.21.15.146:0/416719434 >> v1:172.21.15.146:6789/0 conn(0x7f77740b4360 legacy=0x7f777414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f777b07b640 1 -- 172.21.15.146:0/416719434 shutdown_connections 2024-03-29T11:35:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f777b07b640 1 -- 172.21.15.146:0/416719434 >> 172.21.15.146:0/416719434 conn(0x7f7774066b00 msgr2=0x7f7774066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f777b07b640 1 -- 172.21.15.146:0/416719434 shutdown_connections 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.542+0000 7f777b07b640 1 -- 172.21.15.146:0/416719434 wait complete. 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.543+0000 7f777b07b640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.543+0000 7f777b07b640 10 monclient: monmap: 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.543+0000 7f777b07b640 1 librados: starting msgr at 2024-03-29T11:35:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.543+0000 7f777b07b640 1 librados: starting objecter 2024-03-29T11:35:00.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.543+0000 7f777b07b640 1 Processor -- start 2024-03-29T11:35:00.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.543+0000 7f777b07b640 1 -- start start 2024-03-29T11:35:00.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.543+0000 7f777b07b640 1 librados: setting wanted keys 2024-03-29T11:35:00.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.543+0000 7f777b07b640 1 librados: calling monclient init 2024-03-29T11:35:00.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.543+0000 7f777b07b640 10 monclient: init 2024-03-29T11:35:00.549 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f04d6813040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:35:00.551 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:00.552+0000 7f935f24b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:35:00.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f777b07b640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f777b07b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:00.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f777b07b640 10 monclient(hunting): picked mon.noname-b con 0x7f777409ac40 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:00.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f777b07b640 10 monclient(hunting): picked mon.noname-c con 0x7f77740add50 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f777b07b640 10 monclient(hunting): picked mon.noname-a con 0x7f77740b0f20 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f777b07b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77741d7340 con 0x7f77740b0f20 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f777b07b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7774140a50 con 0x7f777409ac40 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f777b07b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7774140cc0 con 0x7f77740add50 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f777b07b640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f777b07b640 10 monclient(hunting): authenticate will time out at 866.788384s 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f7778e17640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f777409ac40 0x7f77741d77d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:39048/0 (socket says 172.21.15.146:39048) 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f7778e17640 1 -- 172.21.15.146:0/1366763108 learned_addr learned my addr 172.21.15.146:0/1366763108 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 client.?.objecter ms_handle_connect 0x7f777409ac40 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 client.?.objecter ms_handle_connect 0x7f77740add50 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 client.?.objecter ms_handle_connect 0x7f77740b0f20 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448878372 0 0) 0x7f7774140a50 con 0x7f777409ac40 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7758004eb0 con 0x7f777409ac40 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303087264 0 0) 0x7f77741d7340 con 0x7f77740b0f20 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7774140a50 con 0x7f77740b0f20 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837540993 0 0) 0x7f7774140cc0 con 0x7f77740add50 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.544+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77741d7340 con 0x7f77740add50 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 578350238 0 0) 0x7f7758004eb0 con 0x7f777409ac40 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7774140cc0 con 0x7f777409ac40 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f77680019a0 con 0x7f777409ac40 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient(hunting): dump: 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3879448556 0 0) 0x7f7774140a50 con 0x7f77740b0f20 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7758004eb0 con 0x7f77740b0f20 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f776c001f00 con 0x7f77740b0f20 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient(hunting): dump: 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 573170128 0 0) 0x7f77741d7340 con 0x7f77740add50 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7774140a50 con 0x7f77740add50 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f7760001830 con 0x7f77740add50 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient(hunting): dump: 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 398956439 0 0) 0x7f7774140cc0 con 0x7f777409ac40 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 >> v1:172.21.15.37:6790/0 conn(0x7f77740add50 legacy=0x7f77740b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 >> v1:172.21.15.37:6789/0 conn(0x7f77740b0f20 legacy=0x7f77741e1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 monclient: found mon.b 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77741d7ee0 con 0x7f777409ac40 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f777b07b640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f777b07b640 10 monclient: _renew_subs 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f777b07b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f777b07b640 1 -- 172.21.15.146:0/1366763108 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77741e3350 con 0x7f777409ac40 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f777b07b640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f777b07b640 10 monclient: _renew_subs 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f777b07b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f777b07b640 1 -- 172.21.15.146:0/1366763108 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77741e36c0 con 0x7f777409ac40 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f777b07b640 1 librados: init done 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7768001c40 con 0x7f777409ac40 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f7768003a60 con 0x7f777409ac40 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f7772ffd640 4 set_mon_vals no callback set 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 monclient: dump: 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7768004010 con 0x7f777409ac40 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1646199020 0 0) 0x7f7768006690 con 0x7f777409ac40 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 10 client.4129.objecter ms_dispatch 0x7f7774000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:35:00.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:35:00.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:35:00.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.545+0000 7f77717fa640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T11:35:00.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.546+0000 7f777b07b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.546+0000 7f777b07b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.546+0000 7f777b07b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.546+0000 7f777b07b640 1 -- 172.21.15.146:0/1366763108 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7754005080 con 0x7f777409ac40 2024-03-29T11:35:00.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.548+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7768006270 con 0x7f777409ac40 2024-03-29T11:35:00.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.548+0000 7f77717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.548+0000 7f77717fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:00.606 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:00.607+0000 7f935f24b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:35:00.627 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:00.630+0000 7f04d6813040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:35:00.652 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:00.654+0000 7f935f24b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:35:00.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.658+0000 7f777b07b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9d447705-705c-4657-ba3d-5ab095b713a1", "id": 2}] 2024-03-29T11:35:00.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.658+0000 7f777b07b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9d447705-705c-4657-ba3d-5ab095b713a1", "id": 2}] 2024-03-29T11:35:00.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.658+0000 7f777b07b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.658+0000 7f777b07b640 1 -- 172.21.15.146:0/1366763108 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9d447705-705c-4657-ba3d-5ab095b713a1", "id": 2} v 0) v1 -- 0x7f77540058b0 con 0x7f777409ac40 2024-03-29T11:35:00.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.664+0000 7f77717fa640 1 -- 172.21.15.146:0/1366763108 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9d447705-705c-4657-ba3d-5ab095b713a1", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3692347228 0 216599125) 0x7f77680068a0 con 0x7f777409ac40 2024-03-29T11:35:00.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.664+0000 7f77717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9d447705-705c-4657-ba3d-5ab095b713a1", "id": 2}] 2024-03-29T11:35:00.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.664+0000 7f77717fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:00.662 INFO:teuthology.orchestra.run.smithi146.stdout:2 2024-03-29T11:35:00.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.665+0000 7f777b07b640 10 librados: watch_flush enter 2024-03-29T11:35:00.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.665+0000 7f777b07b640 10 librados: watch_flush exit 2024-03-29T11:35:00.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.665+0000 7f777b07b640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T11:35:00.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.665+0000 7f777b07b640 10 client.4129.objecter successfully canceled tick 2024-03-29T11:35:00.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.665+0000 7f777b07b640 10 monclient: shutdown 2024-03-29T11:35:00.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.665+0000 7f777b07b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.665+0000 7f777b07b640 1 -- 172.21.15.146:0/1366763108 >> v1:172.21.15.146:6789/0 conn(0x7f777409ac40 legacy=0x7f77741d77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.665+0000 7f777b07b640 1 -- 172.21.15.146:0/1366763108 shutdown_connections 2024-03-29T11:35:00.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.665+0000 7f777b07b640 1 -- 172.21.15.146:0/1366763108 >> 172.21.15.146:0/1366763108 conn(0x7f7774066b00 msgr2=0x7f777413b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.665+0000 7f777b07b640 1 -- 172.21.15.146:0/1366763108 shutdown_connections 2024-03-29T11:35:00.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.665+0000 7f777b07b640 1 -- 172.21.15.146:0/1366763108 wait complete. 2024-03-29T11:35:00.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.665+0000 7f777b07b640 1 librados: shutdown 2024-03-29T11:35:00.673 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new b73b3c8f-d12d-4c7e-8df9-637a89f5df27 3 2024-03-29T11:35:00.674 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:00.677+0000 7f04d6813040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:35:00.709 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:00.710+0000 7f935f24b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:35:00.732 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:00.734+0000 7f04d6813040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:35:00.753 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:00.754+0000 7f935f24b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:35:00.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.775+0000 7f225a01e640 10 monclient: get_monmap_and_config 2024-03-29T11:35:00.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.775+0000 7f225a01e640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.776+0000 7f225a01e640 10 monclient: monmap: 2024-03-29T11:35:00.773 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:00.773 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.773 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:35:00.776705+0000 2024-03-29T11:35:00.773 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:35:00.776705+0000 2024-03-29T11:35:00.774 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.774 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.774 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:00.774 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:00.774 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:00.774 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.776+0000 7f225a01e640 1 Processor -- start 2024-03-29T11:35:00.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.776+0000 7f225a01e640 1 -- start start 2024-03-29T11:35:00.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.776+0000 7f225a01e640 10 monclient: init 2024-03-29T11:35:00.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f225a01e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f225a01e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:00.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f225a01e640 10 monclient(hunting): picked mon.noname-b con 0x7f225409ac40 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:00.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f225a01e640 10 monclient(hunting): picked mon.noname-c con 0x7f225409c870 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:00.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f225a01e640 10 monclient(hunting): picked mon.noname-a con 0x7f22540b4360 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:00.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f225a01e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225409dec0 con 0x7f22540b4360 2024-03-29T11:35:00.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f225a01e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225409da30 con 0x7f225409c870 2024-03-29T11:35:00.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f225a01e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22540b39c0 con 0x7f225409ac40 2024-03-29T11:35:00.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f225a01e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f225a01e640 10 monclient(hunting): authenticate will time out at 867.021250s 2024-03-29T11:35:00.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f2252ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f225409c870 0x7f225413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:39054/0 (socket says 172.21.15.146:39054) 2024-03-29T11:35:00.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f2252ffd640 1 -- 172.21.15.146:0/373157869 learned_addr learned my addr 172.21.15.146:0/373157869 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:00.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 1 -- 172.21.15.146:0/373157869 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084453644 0 0) 0x7f225409da30 con 0x7f225409c870 2024-03-29T11:35:00.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 1 -- 172.21.15.146:0/373157869 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f223c003520 con 0x7f225409c870 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 1 -- 172.21.15.146:0/373157869 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744890612 0 0) 0x7f22540b39c0 con 0x7f225409ac40 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 1 -- 172.21.15.146:0/373157869 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f225409da30 con 0x7f225409ac40 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 1 -- 172.21.15.146:0/373157869 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186470542 0 0) 0x7f224c001490 con 0x7f22540b4360 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:35:00.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 1 -- 172.21.15.146:0/373157869 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22540b39c0 con 0x7f22540b4360 2024-03-29T11:35:00.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 1 -- 172.21.15.146:0/373157869 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 681184569 0 0) 0x7f223c003520 con 0x7f225409c870 2024-03-29T11:35:00.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 1 -- 172.21.15.146:0/373157869 >> v1:172.21.15.37:6790/0 conn(0x7f225409ac40 legacy=0x7f22540b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 1 -- 172.21.15.146:0/373157869 >> v1:172.21.15.37:6789/0 conn(0x7f22540b4360 legacy=0x7f225414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 1 monclient: found mon.noname-c 2024-03-29T11:35:00.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.777+0000 7f22527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 1 -- 172.21.15.146:0/373157869 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22540b5160 con 0x7f225409c870 2024-03-29T11:35:00.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 1 -- 172.21.15.146:0/373157869 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2248001690 con 0x7f225409c870 2024-03-29T11:35:00.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f225a01e640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:35:00.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f225a01e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 10 monclient: dump: 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f225a01e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:00.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 1 -- 172.21.15.146:0/373157869 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22480017f0 con 0x7f225409c870 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 1 -- 172.21.15.146:0/373157869 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2248001c00 con 0x7f225409c870 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f225a01e640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f225a01e640 4 set_mon_vals no callback set 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f22527fc640 10 monclient: dump: 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f225a01e640 10 monclient: shutdown 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f225a01e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f225a01e640 1 -- 172.21.15.146:0/373157869 >> v1:172.21.15.146:6789/0 conn(0x7f225409c870 legacy=0x7f225413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f225a01e640 1 -- 172.21.15.146:0/373157869 shutdown_connections 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f225a01e640 1 -- 172.21.15.146:0/373157869 >> 172.21.15.146:0/373157869 conn(0x7f2254066b00 msgr2=0x7f2254066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f225a01e640 1 -- 172.21.15.146:0/373157869 shutdown_connections 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.778+0000 7f225a01e640 1 -- 172.21.15.146:0/373157869 wait complete. 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.779+0000 7f225a01e640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.779+0000 7f225a01e640 10 monclient: monmap: 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-29T11:35:00.784 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-29T11:35:00.785 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.785 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.785 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:00.785 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:00.785 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:00.785 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.779+0000 7f225a01e640 1 librados: starting msgr at 2024-03-29T11:35:00.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.779+0000 7f225a01e640 1 librados: starting objecter 2024-03-29T11:35:00.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.779+0000 7f225a01e640 1 Processor -- start 2024-03-29T11:35:00.785 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:00.779+0000 7f04d6813040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:35:00.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.779+0000 7f225a01e640 1 -- start start 2024-03-29T11:35:00.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.779+0000 7f225a01e640 1 librados: setting wanted keys 2024-03-29T11:35:00.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.779+0000 7f225a01e640 1 librados: calling monclient init 2024-03-29T11:35:00.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.779+0000 7f225a01e640 10 monclient: init 2024-03-29T11:35:00.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f225a01e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f225a01e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:00.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f225a01e640 10 monclient(hunting): picked mon.noname-b con 0x7f225409ac40 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:00.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f225a01e640 10 monclient(hunting): picked mon.noname-a con 0x7f22541391f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:00.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f225a01e640 10 monclient(hunting): picked mon.noname-c con 0x7f22540b0330 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:00.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f225a01e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22541dba50 con 0x7f22541391f0 2024-03-29T11:35:00.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f225a01e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22541db770 con 0x7f225409ac40 2024-03-29T11:35:00.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f225a01e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22541db4a0 con 0x7f22540b0330 2024-03-29T11:35:00.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f225a01e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f225a01e640 10 monclient(hunting): authenticate will time out at 867.024327s 2024-03-29T11:35:00.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f22537fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f225409ac40 0x7f22541dad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:39062/0 (socket says 172.21.15.146:39062) 2024-03-29T11:35:00.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f22537fe640 1 -- 172.21.15.146:0/3223968183 learned_addr learned my addr 172.21.15.146:0/3223968183 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 client.?.objecter ms_handle_connect 0x7f225409ac40 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 client.?.objecter ms_handle_connect 0x7f22541391f0 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 client.?.objecter ms_handle_connect 0x7f22540b0330 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157536019 0 0) 0x7f22541db770 con 0x7f225409ac40 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2240004eb0 con 0x7f225409ac40 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645156038 0 0) 0x7f22541dba50 con 0x7f22541391f0 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22541db770 con 0x7f22541391f0 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510928016 0 0) 0x7f22541db4a0 con 0x7f22540b0330 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.780+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22541dba50 con 0x7f22540b0330 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1114529997 0 0) 0x7f22541db770 con 0x7f22541391f0 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22541db4a0 con 0x7f22541391f0 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4112992972 0 0) 0x7f2240004eb0 con 0x7f225409ac40 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22541db770 con 0x7f225409ac40 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2248001b00 con 0x7f22541391f0 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient(hunting): dump: 2024-03-29T11:35:00.794 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.794 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.794 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.794 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.794 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.794 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.794 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.794 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.794 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.794 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:00.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2244001a30 con 0x7f225409ac40 2024-03-29T11:35:00.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient(hunting): dump: 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3808097902 0 0) 0x7f22541dba50 con 0x7f22540b0330 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2240004eb0 con 0x7f22540b0330 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f224c001810 con 0x7f22540b0330 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient(hunting): dump: 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3407462720 0 0) 0x7f22541db4a0 con 0x7f22541391f0 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 >> v1:172.21.15.37:6790/0 conn(0x7f22540b0330 legacy=0x7f22541e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 >> v1:172.21.15.146:6789/0 conn(0x7f225409ac40 legacy=0x7f22541dad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 monclient: found mon.a 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22540b07c0 con 0x7f22541391f0 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f225a01e640 5 monclient: authenticate success, global_id 4134 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f225a01e640 10 monclient: _renew_subs 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f225a01e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f225a01e640 1 -- 172.21.15.146:0/3223968183 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22541e37a0 con 0x7f22541391f0 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f225a01e640 10 client.4134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f225a01e640 10 monclient: _renew_subs 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f225a01e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f225a01e640 1 -- 172.21.15.146:0/3223968183 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22541e3b10 con 0x7f22541391f0 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f225a01e640 1 librados: init done 2024-03-29T11:35:00.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2248001210 con 0x7f22541391f0 2024-03-29T11:35:00.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2248004c30 con 0x7f22541391f0 2024-03-29T11:35:00.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:00.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f2251ffb640 4 set_mon_vals no callback set 2024-03-29T11:35:00.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 monclient: dump: 2024-03-29T11:35:00.798 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.798 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.798 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.798 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.798 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.798 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.798 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.798 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.799 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.799 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f22480051a0 con 0x7f22541391f0 2024-03-29T11:35:00.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 4169152363 0 0) 0x7f2248006810 con 0x7f22541391f0 2024-03-29T11:35:00.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 10 client.4134.objecter ms_dispatch 0x7f2254000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:35:00.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 3 client.4134.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:35:00.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 3 client.4134.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:35:00.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.781+0000 7f223bfff640 20 client.4134.objecter dump_active .. 0 homeless 2024-03-29T11:35:00.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.782+0000 7f225a01e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.782+0000 7f225a01e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.782+0000 7f225a01e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:00.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.782+0000 7f225a01e640 1 -- 172.21.15.146:0/3223968183 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2234005080 con 0x7f22541391f0 2024-03-29T11:35:00.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.784+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2248006430 con 0x7f22541391f0 2024-03-29T11:35:00.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.784+0000 7f223bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.784+0000 7f223bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:00.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.856+0000 7f225a01e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b73b3c8f-d12d-4c7e-8df9-637a89f5df27", "id": 3}] 2024-03-29T11:35:00.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.856+0000 7f225a01e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b73b3c8f-d12d-4c7e-8df9-637a89f5df27", "id": 3}] 2024-03-29T11:35:00.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.856+0000 7f225a01e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:00.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.856+0000 7f225a01e640 1 -- 172.21.15.146:0/3223968183 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b73b3c8f-d12d-4c7e-8df9-637a89f5df27", "id": 3} v 0) v1 -- 0x7f22340058b0 con 0x7f22541391f0 2024-03-29T11:35:00.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.858+0000 7f223bfff640 1 -- 172.21.15.146:0/3223968183 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b73b3c8f-d12d-4c7e-8df9-637a89f5df27", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 696892766 0 525046306) 0x7f2248006a40 con 0x7f22541391f0 2024-03-29T11:35:00.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.858+0000 7f223bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b73b3c8f-d12d-4c7e-8df9-637a89f5df27", "id": 3}] 2024-03-29T11:35:00.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.858+0000 7f223bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:00.856 INFO:teuthology.orchestra.run.smithi146.stdout:3 2024-03-29T11:35:00.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.859+0000 7f225a01e640 10 librados: watch_flush enter 2024-03-29T11:35:00.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.859+0000 7f225a01e640 10 librados: watch_flush exit 2024-03-29T11:35:00.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.859+0000 7f225a01e640 20 client.4134.objecter shutdown clearing up homeless session... 2024-03-29T11:35:00.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.859+0000 7f225a01e640 10 client.4134.objecter successfully canceled tick 2024-03-29T11:35:00.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.859+0000 7f225a01e640 10 monclient: shutdown 2024-03-29T11:35:00.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.859+0000 7f225a01e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.859+0000 7f225a01e640 1 -- 172.21.15.146:0/3223968183 >> v1:172.21.15.37:6789/0 conn(0x7f22541391f0 legacy=0x7f22540afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.859+0000 7f225a01e640 1 -- 172.21.15.146:0/3223968183 shutdown_connections 2024-03-29T11:35:00.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.859+0000 7f225a01e640 1 -- 172.21.15.146:0/3223968183 >> 172.21.15.146:0/3223968183 conn(0x7f2254066b00 msgr2=0x7f225413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.859+0000 7f225a01e640 1 -- 172.21.15.146:0/3223968183 shutdown_connections 2024-03-29T11:35:00.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.859+0000 7f225a01e640 1 -- 172.21.15.146:0/3223968183 wait complete. 2024-03-29T11:35:00.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.859+0000 7f225a01e640 1 librados: shutdown 2024-03-29T11:35:00.867 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad 4 2024-03-29T11:35:00.873 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:00.875+0000 7f935f24b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:35:00.899 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:00.902+0000 7f04d6813040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:35:00.946 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:00.947+0000 7f935f24b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:35:00.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.968+0000 7fa0ca1a3640 10 monclient: get_monmap_and_config 2024-03-29T11:35:00.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.968+0000 7fa0ca1a3640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.968+0000 7fa0ca1a3640 10 monclient: monmap: 2024-03-29T11:35:00.966 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:00.966 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.966 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:35:00.969483+0000 2024-03-29T11:35:00.966 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:35:00.969483+0000 2024-03-29T11:35:00.966 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.967 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.967 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:00.967 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:00.967 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:00.967 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.969+0000 7fa0ca1a3640 1 Processor -- start 2024-03-29T11:35:00.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.969+0000 7fa0ca1a3640 1 -- start start 2024-03-29T11:35:00.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.969+0000 7fa0ca1a3640 10 monclient: init 2024-03-29T11:35:00.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0ca1a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0ca1a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:00.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0ca1a3640 10 monclient(hunting): picked mon.noname-a con 0x7fa0c409ac40 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:00.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0ca1a3640 10 monclient(hunting): picked mon.noname-c con 0x7fa0c409c870 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:00.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0ca1a3640 10 monclient(hunting): picked mon.noname-b con 0x7fa0c40b4360 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:00.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0ca1a3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c409dec0 con 0x7fa0c409ac40 2024-03-29T11:35:00.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0ca1a3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c409da30 con 0x7fa0c409c870 2024-03-29T11:35:00.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0ca1a3640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c40b39c0 con 0x7fa0c40b4360 2024-03-29T11:35:00.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0ca1a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0ca1a3640 10 monclient(hunting): authenticate will time out at 867.214287s 2024-03-29T11:35:00.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0bbfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa0c409c870 0x7fa0c413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:39076/0 (socket says 172.21.15.146:39076) 2024-03-29T11:35:00.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0bbfff640 1 -- 172.21.15.146:0/4020785395 learned_addr learned my addr 172.21.15.146:0/4020785395 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:00.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 1 -- 172.21.15.146:0/4020785395 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739366693 0 0) 0x7fa0c409da30 con 0x7fa0c409c870 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 1 -- 172.21.15.146:0/4020785395 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b0003520 con 0x7fa0c409c870 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 1 -- 172.21.15.146:0/4020785395 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983490101 0 0) 0x7fa0b4001110 con 0x7fa0c409ac40 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 1 -- 172.21.15.146:0/4020785395 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0c409da30 con 0x7fa0c409ac40 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 1 -- 172.21.15.146:0/4020785395 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362941661 0 0) 0x7fa0c40b39c0 con 0x7fa0c40b4360 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:35:00.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.970+0000 7fa0c2ffd640 1 -- 172.21.15.146:0/4020785395 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b4001110 con 0x7fa0c40b4360 2024-03-29T11:35:00.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 1 -- 172.21.15.146:0/4020785395 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 342657393 0 0) 0x7fa0b0003520 con 0x7fa0c409c870 2024-03-29T11:35:00.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 1 -- 172.21.15.146:0/4020785395 >> v1:172.21.15.37:6790/0 conn(0x7fa0c40b4360 legacy=0x7fa0c414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 1 -- 172.21.15.146:0/4020785395 >> v1:172.21.15.37:6789/0 conn(0x7fa0c409ac40 legacy=0x7fa0c40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 1 monclient: found mon.noname-c 2024-03-29T11:35:00.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 1 -- 172.21.15.146:0/4020785395 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c40b5160 con 0x7fa0c409c870 2024-03-29T11:35:00.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 1 -- 172.21.15.146:0/4020785395 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa0ac001690 con 0x7fa0c409c870 2024-03-29T11:35:00.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0ca1a3640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:35:00.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0ca1a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:00.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:00.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 10 monclient: dump: 2024-03-29T11:35:00.979 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.979 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.979 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.979 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.980 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.980 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.980 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.980 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.980 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.980 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0ca1a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:00.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 1 -- 172.21.15.146:0/4020785395 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0ac001870 con 0x7fa0c409c870 2024-03-29T11:35:00.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 1 -- 172.21.15.146:0/4020785395 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa0ac001c80 con 0x7fa0c409c870 2024-03-29T11:35:00.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0ca1a3640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:00.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0ca1a3640 4 set_mon_vals no callback set 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0c2ffd640 10 monclient: dump: 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0ca1a3640 10 monclient: shutdown 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0ca1a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0ca1a3640 1 -- 172.21.15.146:0/4020785395 >> v1:172.21.15.146:6789/0 conn(0x7fa0c409c870 legacy=0x7fa0c413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0ca1a3640 1 -- 172.21.15.146:0/4020785395 shutdown_connections 2024-03-29T11:35:00.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0ca1a3640 1 -- 172.21.15.146:0/4020785395 >> 172.21.15.146:0/4020785395 conn(0x7fa0c4066b00 msgr2=0x7fa0c4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0ca1a3640 1 -- 172.21.15.146:0/4020785395 shutdown_connections 2024-03-29T11:35:00.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.971+0000 7fa0ca1a3640 1 -- 172.21.15.146:0/4020785395 wait complete. 2024-03-29T11:35:00.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.972+0000 7fa0ca1a3640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.972+0000 7fa0ca1a3640 10 monclient: monmap: 2024-03-29T11:35:00.982 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:00.982 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:00.982 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-29T11:35:00.982 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-29T11:35:00.982 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.982 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.982 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.972+0000 7fa0ca1a3640 1 librados: starting msgr at 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.972+0000 7fa0ca1a3640 1 librados: starting objecter 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.972+0000 7fa0ca1a3640 1 Processor -- start 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.972+0000 7fa0ca1a3640 1 -- start start 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.972+0000 7fa0ca1a3640 1 librados: setting wanted keys 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.972+0000 7fa0ca1a3640 1 librados: calling monclient init 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.972+0000 7fa0ca1a3640 10 monclient: init 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0ca1a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0ca1a3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0ca1a3640 10 monclient(hunting): picked mon.noname-a con 0x7fa0c409ac40 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0ca1a3640 10 monclient(hunting): picked mon.noname-c con 0x7fa0c40b7eb0 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0ca1a3640 10 monclient(hunting): picked mon.noname-b con 0x7fa0c413ffa0 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:00.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0ca1a3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c4143060 con 0x7fa0c409ac40 2024-03-29T11:35:00.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0ca1a3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c41d7760 con 0x7fa0c413ffa0 2024-03-29T11:35:00.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0ca1a3640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c41d78f0 con 0x7fa0c40b7eb0 2024-03-29T11:35:00.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0ca1a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0ca1a3640 10 monclient(hunting): authenticate will time out at 867.217402s 2024-03-29T11:35:00.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c3fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa0c413ffa0 0x7fa0c41e1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:39086/0 (socket says 172.21.15.146:39086) 2024-03-29T11:35:00.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c3fff640 1 -- 172.21.15.146:0/1190565793 learned_addr learned my addr 172.21.15.146:0/1190565793 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:00.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 client.?.objecter ms_handle_connect 0x7fa0c413ffa0 2024-03-29T11:35:00.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 client.?.objecter ms_handle_connect 0x7fa0c409ac40 2024-03-29T11:35:00.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 client.?.objecter ms_handle_connect 0x7fa0c40b7eb0 2024-03-29T11:35:00.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294743461 0 0) 0x7fa0c41d7760 con 0x7fa0c413ffa0 2024-03-29T11:35:00.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0a4004eb0 con 0x7fa0c413ffa0 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485081399 0 0) 0x7fa0c4143060 con 0x7fa0c409ac40 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0c41d7760 con 0x7fa0c409ac40 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355762250 0 0) 0x7fa0c41d78f0 con 0x7fa0c40b7eb0 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:35:00.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0c4143060 con 0x7fa0c40b7eb0 2024-03-29T11:35:00.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.973+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 395404894 0 0) 0x7fa0a4004eb0 con 0x7fa0c413ffa0 2024-03-29T11:35:00.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0c41d78f0 con 0x7fa0c413ffa0 2024-03-29T11:35:00.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa0bc0042d0 con 0x7fa0c413ffa0 2024-03-29T11:35:00.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:00.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:00.986 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.986 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.986 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.986 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.986 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.986 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.986 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3878952077 0 0) 0x7fa0c41d7760 con 0x7fa0c409ac40 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0a4004eb0 con 0x7fa0c409ac40 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa0b40018e0 con 0x7fa0c409ac40 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.987 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3385138168 0 0) 0x7fa0c4143060 con 0x7fa0c40b7eb0 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0c41d7760 con 0x7fa0c40b7eb0 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa0ac0019e0 con 0x7fa0c40b7eb0 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.988 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 33240911 0 0) 0x7fa0c41d78f0 con 0x7fa0c413ffa0 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 >> v1:172.21.15.37:6790/0 conn(0x7fa0c40b7eb0 legacy=0x7fa0c41d7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 >> v1:172.21.15.37:6789/0 conn(0x7fa0c409ac40 legacy=0x7fa0c41427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 monclient: found mon.b 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c4140460 con 0x7fa0c413ffa0 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0ca1a3640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:35:00.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0ca1a3640 10 monclient: _renew_subs 2024-03-29T11:35:00.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0ca1a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0ca1a3640 1 -- 172.21.15.146:0/1190565793 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c4140710 con 0x7fa0c413ffa0 2024-03-29T11:35:00.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0ca1a3640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:00.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0ca1a3640 10 monclient: _renew_subs 2024-03-29T11:35:00.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0bc001250 con 0x7fa0c413ffa0 2024-03-29T11:35:00.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0ca1a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0ca1a3640 1 -- 172.21.15.146:0/1190565793 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c41e3390 con 0x7fa0c413ffa0 2024-03-29T11:35:00.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0ca1a3640 1 librados: init done 2024-03-29T11:35:00.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa0bc004840 con 0x7fa0c413ffa0 2024-03-29T11:35:00.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c27fc640 4 set_mon_vals no callback set 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 monclient: dump: 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa0bc001a30 con 0x7fa0c413ffa0 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 4290271396 0 0) 0x7fa0bc0069f0 con 0x7fa0c413ffa0 2024-03-29T11:35:00.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 10 client.4141.objecter ms_dispatch 0x7fa0c4000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:35:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:35:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:35:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.974+0000 7fa0c0ff9640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:35:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.975+0000 7fa0ca1a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.975+0000 7fa0ca1a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.975+0000 7fa0ca1a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.975+0000 7fa0ca1a3640 1 -- 172.21.15.146:0/1190565793 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c409b020 con 0x7fa0c413ffa0 2024-03-29T11:35:00.992 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:00.976+0000 7f04d6813040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:35:00.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.977+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa0bc001f10 con 0x7fa0c413ffa0 2024-03-29T11:35:00.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.977+0000 7fa0c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:00.977+0000 7fa0c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:00.994 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:00.996+0000 7f935f24b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:35:01.021 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:01.023+0000 7f04d6813040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:35:01.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.048+0000 7fa0ca1a3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad", "id": 4}] 2024-03-29T11:35:01.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.048+0000 7fa0ca1a3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad", "id": 4}] 2024-03-29T11:35:01.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.048+0000 7fa0ca1a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.049+0000 7fa0ca1a3640 1 -- 172.21.15.146:0/1190565793 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad", "id": 4} v 0) v1 -- 0x7fa0c404f730 con 0x7fa0c413ffa0 2024-03-29T11:35:01.048 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:01.050+0000 7f935f24b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:35:01.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.052+0000 7fa0c0ff9640 1 -- 172.21.15.146:0/1190565793 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2343773598 0 1697012583) 0x7fa0bc0064e0 con 0x7fa0c413ffa0 2024-03-29T11:35:01.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.052+0000 7fa0c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad", "id": 4}] 2024-03-29T11:35:01.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.052+0000 7fa0c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:01.050 INFO:teuthology.orchestra.run.smithi146.stdout:4 2024-03-29T11:35:01.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.053+0000 7fa0ca1a3640 10 librados: watch_flush enter 2024-03-29T11:35:01.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.053+0000 7fa0ca1a3640 10 librados: watch_flush exit 2024-03-29T11:35:01.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.053+0000 7fa0ca1a3640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:35:01.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.053+0000 7fa0ca1a3640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:35:01.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.053+0000 7fa0ca1a3640 10 monclient: shutdown 2024-03-29T11:35:01.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.053+0000 7fa0ca1a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.053+0000 7fa0ca1a3640 1 -- 172.21.15.146:0/1190565793 >> v1:172.21.15.146:6789/0 conn(0x7fa0c413ffa0 legacy=0x7fa0c41e1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.053+0000 7fa0ca1a3640 1 -- 172.21.15.146:0/1190565793 shutdown_connections 2024-03-29T11:35:01.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.053+0000 7fa0ca1a3640 1 -- 172.21.15.146:0/1190565793 >> 172.21.15.146:0/1190565793 conn(0x7fa0c4066b00 msgr2=0x7fa0c413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.054+0000 7fa0ca1a3640 1 -- 172.21.15.146:0/1190565793 shutdown_connections 2024-03-29T11:35:01.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.054+0000 7fa0ca1a3640 1 -- 172.21.15.146:0/1190565793 wait complete. 2024-03-29T11:35:01.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.054+0000 7fa0ca1a3640 1 librados: shutdown 2024-03-29T11:35:01.061 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 8adc2525-35e0-477a-8946-2a6afe773016 5 2024-03-29T11:35:01.076 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:01.078+0000 7f04d6813040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:35:01.097 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:01.099+0000 7f935f24b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:35:01.126 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:01.128+0000 7f04d6813040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:35:01.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.177+0000 7f2aca73b640 10 monclient: get_monmap_and_config 2024-03-29T11:35:01.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.177+0000 7f2aca73b640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.177+0000 7f2aca73b640 10 monclient: monmap: 2024-03-29T11:35:01.175 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:01.175 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.175 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:35:01.178013+0000 2024-03-29T11:35:01.175 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:35:01.178013+0000 2024-03-29T11:35:01.175 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.175 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.175 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:01.175 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:01.175 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:01.175 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2aca73b640 1 Processor -- start 2024-03-29T11:35:01.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2aca73b640 1 -- start start 2024-03-29T11:35:01.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2aca73b640 10 monclient: init 2024-03-29T11:35:01.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2aca73b640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2aca73b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:01.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2aca73b640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac40af360 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:01.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2aca73b640 10 monclient(hunting): picked mon.noname-a con 0x7f2ac406af60 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:01.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2aca73b640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac4145e60 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:01.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2aca73b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac40b0520 con 0x7f2ac406af60 2024-03-29T11:35:01.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2aca73b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac40aff20 con 0x7f2ac40af360 2024-03-29T11:35:01.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2aca73b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac40b1da0 con 0x7f2ac4145e60 2024-03-29T11:35:01.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2aca73b640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2aca73b640 10 monclient(hunting): authenticate will time out at 867.422645s 2024-03-29T11:35:01.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2ac3fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2ac40af360 0x7f2ac406c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:39088/0 (socket says 172.21.15.146:39088) 2024-03-29T11:35:01.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.178+0000 7f2ac3fff640 1 -- 172.21.15.146:0/2871706604 learned_addr learned my addr 172.21.15.146:0/2871706604 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:01.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 1 -- 172.21.15.146:0/2871706604 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140871316 0 0) 0x7f2ac40aff20 con 0x7f2ac40af360 2024-03-29T11:35:01.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 1 -- 172.21.15.146:0/2871706604 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aac003520 con 0x7f2ac40af360 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 1 -- 172.21.15.146:0/2871706604 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093345712 0 0) 0x7f2ac40b0520 con 0x7f2ac406af60 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 1 -- 172.21.15.146:0/2871706604 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac40aff20 con 0x7f2ac406af60 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 1 -- 172.21.15.146:0/2871706604 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992404555 0 0) 0x7f2ac40b1da0 con 0x7f2ac4145e60 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:35:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 1 -- 172.21.15.146:0/2871706604 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac40b0520 con 0x7f2ac4145e60 2024-03-29T11:35:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 1 -- 172.21.15.146:0/2871706604 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1988422687 0 0) 0x7f2aac003520 con 0x7f2ac40af360 2024-03-29T11:35:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 1 -- 172.21.15.146:0/2871706604 >> v1:172.21.15.37:6790/0 conn(0x7f2ac4145e60 legacy=0x7f2ac414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 1 -- 172.21.15.146:0/2871706604 >> v1:172.21.15.37:6789/0 conn(0x7f2ac406af60 legacy=0x7f2ac4145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 1 monclient: found mon.noname-c 2024-03-29T11:35:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 1 -- 172.21.15.146:0/2871706604 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac40b2080 con 0x7f2ac40af360 2024-03-29T11:35:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 1 -- 172.21.15.146:0/2871706604 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2ab4001690 con 0x7f2ac40af360 2024-03-29T11:35:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2aca73b640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2aca73b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient: dump: 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2aca73b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 1 -- 172.21.15.146:0/2871706604 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ab4001870 con 0x7f2ac40af360 2024-03-29T11:35:01.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2aca73b640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 1 -- 172.21.15.146:0/2871706604 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2ab4001c00 con 0x7f2ac40af360 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2aca73b640 4 set_mon_vals no callback set 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.179+0000 7f2ac2ffd640 10 monclient: dump: 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.188 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.180+0000 7f2aca73b640 10 monclient: shutdown 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.180+0000 7f2aca73b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.180+0000 7f2aca73b640 1 -- 172.21.15.146:0/2871706604 >> v1:172.21.15.146:6789/0 conn(0x7f2ac40af360 legacy=0x7f2ac406c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.180+0000 7f2aca73b640 1 -- 172.21.15.146:0/2871706604 shutdown_connections 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.180+0000 7f2aca73b640 1 -- 172.21.15.146:0/2871706604 >> 172.21.15.146:0/2871706604 conn(0x7f2ac40b58f0 msgr2=0x7f2ac40b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.180+0000 7f2aca73b640 1 -- 172.21.15.146:0/2871706604 shutdown_connections 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.180+0000 7f2aca73b640 1 -- 172.21.15.146:0/2871706604 wait complete. 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.180+0000 7f2aca73b640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.180+0000 7f2aca73b640 10 monclient: monmap: 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:01.190 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.180+0000 7f2aca73b640 1 librados: starting msgr at 2024-03-29T11:35:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.180+0000 7f2aca73b640 1 librados: starting objecter 2024-03-29T11:35:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 1 Processor -- start 2024-03-29T11:35:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 1 -- start start 2024-03-29T11:35:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 1 librados: setting wanted keys 2024-03-29T11:35:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 1 librados: calling monclient init 2024-03-29T11:35:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 10 monclient: init 2024-03-29T11:35:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 10 monclient(hunting): picked mon.noname-a con 0x7f2ac406af60 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac40af360 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac4145e60 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac41214f0 con 0x7f2ac406af60 2024-03-29T11:35:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac41217d0 con 0x7f2ac4145e60 2024-03-29T11:35:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac41db170 con 0x7f2ac40af360 2024-03-29T11:35:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2aca73b640 10 monclient(hunting): authenticate will time out at 867.425690s 2024-03-29T11:35:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2ac8cd8640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2ac4145e60 0x7f2ac41d6ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:39104/0 (socket says 172.21.15.146:39104) 2024-03-29T11:35:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2ac8cd8640 1 -- 172.21.15.146:0/2600436507 learned_addr learned my addr 172.21.15.146:0/2600436507 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2ac0ff9640 10 client.?.objecter ms_handle_connect 0x7f2ac4145e60 2024-03-29T11:35:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2ac0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2ac0ff9640 10 client.?.objecter ms_handle_connect 0x7f2ac40af360 2024-03-29T11:35:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.181+0000 7f2ac0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 client.?.objecter ms_handle_connect 0x7f2ac406af60 2024-03-29T11:35:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527828134 0 0) 0x7f2ac41217d0 con 0x7f2ac4145e60 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab0004eb0 con 0x7f2ac4145e60 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318350018 0 0) 0x7f2ac41db170 con 0x7f2ac40af360 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac41217d0 con 0x7f2ac40af360 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748599931 0 0) 0x7f2ac41214f0 con 0x7f2ac406af60 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac41db170 con 0x7f2ac406af60 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3672324882 0 0) 0x7f2ab0004eb0 con 0x7f2ac4145e60 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ac41214f0 con 0x7f2ac4145e60 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2abc001c10 con 0x7f2ac4145e60 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.194 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2530619196 0 0) 0x7f2ac41db170 con 0x7f2ac406af60 2024-03-29T11:35:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ab0004eb0 con 0x7f2ac406af60 2024-03-29T11:35:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2ab4001b80 con 0x7f2ac406af60 2024-03-29T11:35:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.195 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1454075062 0 0) 0x7f2ac41217d0 con 0x7f2ac40af360 2024-03-29T11:35:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ac41db170 con 0x7f2ac40af360 2024-03-29T11:35:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2ab8001930 con 0x7f2ac40af360 2024-03-29T11:35:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.196 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 731213939 0 0) 0x7f2ac41214f0 con 0x7f2ac4145e60 2024-03-29T11:35:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 >> v1:172.21.15.37:6790/0 conn(0x7f2ac40af360 legacy=0x7f2ac413e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 >> v1:172.21.15.37:6789/0 conn(0x7f2ac406af60 legacy=0x7f2ac41da930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 monclient: found mon.b 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac40719e0 con 0x7f2ac4145e60 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2aca73b640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2aca73b640 10 monclient: _renew_subs 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2aca73b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2aca73b640 1 -- 172.21.15.146:0/2600436507 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac4071c90 con 0x7f2ac4145e60 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2aca73b640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2aca73b640 10 monclient: _renew_subs 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2aca73b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2aca73b640 1 -- 172.21.15.146:0/2600436507 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac413ea60 con 0x7f2ac4145e60 2024-03-29T11:35:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2aca73b640 1 librados: init done 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2abc003990 con 0x7f2ac4145e60 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2abc003d60 con 0x7f2ac4145e60 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 monclient: dump: 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.198 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac27fc640 4 set_mon_vals no callback set 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2abc004310 con 0x7f2ac4145e60 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2929164471 0 0) 0x7f2abc006ab0 con 0x7f2ac4145e60 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 10 client.4147.objecter ms_dispatch 0x7f2ac4000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.182+0000 7f2ac0ff9640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.183+0000 7f2aca73b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.183+0000 7f2aca73b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.183+0000 7f2aca73b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.183+0000 7f2aca73b640 1 -- 172.21.15.146:0/2600436507 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa4005080 con 0x7f2ac4145e60 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.185+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2abc006530 con 0x7f2ac4145e60 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.185+0000 7f2ac0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.185+0000 7f2ac0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:01.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.256+0000 7f2aca73b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8adc2525-35e0-477a-8946-2a6afe773016", "id": 5}] 2024-03-29T11:35:01.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.256+0000 7f2aca73b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8adc2525-35e0-477a-8946-2a6afe773016", "id": 5}] 2024-03-29T11:35:01.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.256+0000 7f2aca73b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.256+0000 7f2aca73b640 1 -- 172.21.15.146:0/2600436507 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8adc2525-35e0-477a-8946-2a6afe773016", "id": 5} v 0) v1 -- 0x7f2aa40058b0 con 0x7f2ac4145e60 2024-03-29T11:35:01.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.260+0000 7f2ac0ff9640 1 -- 172.21.15.146:0/2600436507 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8adc2525-35e0-477a-8946-2a6afe773016", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1934463383 0 1988412176) 0x7f2abc006750 con 0x7f2ac4145e60 2024-03-29T11:35:01.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.260+0000 7f2ac0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8adc2525-35e0-477a-8946-2a6afe773016", "id": 5}] 2024-03-29T11:35:01.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.260+0000 7f2ac0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:01.258 INFO:teuthology.orchestra.run.smithi146.stdout:5 2024-03-29T11:35:01.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.261+0000 7f2aca73b640 10 librados: watch_flush enter 2024-03-29T11:35:01.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.261+0000 7f2aca73b640 10 librados: watch_flush exit 2024-03-29T11:35:01.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.261+0000 7f2aca73b640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T11:35:01.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.261+0000 7f2aca73b640 10 client.4147.objecter successfully canceled tick 2024-03-29T11:35:01.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.261+0000 7f2aca73b640 10 monclient: shutdown 2024-03-29T11:35:01.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.261+0000 7f2aca73b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.261+0000 7f2aca73b640 1 -- 172.21.15.146:0/2600436507 >> v1:172.21.15.146:6789/0 conn(0x7f2ac4145e60 legacy=0x7f2ac41d6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.261+0000 7f2aca73b640 1 -- 172.21.15.146:0/2600436507 shutdown_connections 2024-03-29T11:35:01.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.261+0000 7f2aca73b640 1 -- 172.21.15.146:0/2600436507 >> 172.21.15.146:0/2600436507 conn(0x7f2ac40b58f0 msgr2=0x7f2ac40b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.261+0000 7f2aca73b640 1 -- 172.21.15.146:0/2600436507 shutdown_connections 2024-03-29T11:35:01.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.261+0000 7f2aca73b640 1 -- 172.21.15.146:0/2600436507 wait complete. 2024-03-29T11:35:01.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.261+0000 7f2aca73b640 1 librados: shutdown 2024-03-29T11:35:01.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 634a1b82-a618-42c8-b306-b42760f2e055 6 2024-03-29T11:35:01.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.372+0000 7f3eafdcb640 10 monclient: get_monmap_and_config 2024-03-29T11:35:01.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.372+0000 7f3eafdcb640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.373+0000 7f3eafdcb640 10 monclient: monmap: 2024-03-29T11:35:01.371 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:01.371 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.371 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:35:01.373695+0000 2024-03-29T11:35:01.371 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:35:01.373695+0000 2024-03-29T11:35:01.371 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.371 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.371 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:01.371 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:01.371 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:01.371 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.373+0000 7f3eafdcb640 1 Processor -- start 2024-03-29T11:35:01.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.373+0000 7f3eafdcb640 1 -- start start 2024-03-29T11:35:01.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.373+0000 7f3eafdcb640 10 monclient: init 2024-03-29T11:35:01.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eafdcb640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eafdcb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:01.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eafdcb640 10 monclient(hunting): picked mon.noname-a con 0x7f3ea809ac40 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:01.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eafdcb640 10 monclient(hunting): picked mon.noname-b con 0x7f3ea809c870 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:01.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eafdcb640 10 monclient(hunting): picked mon.noname-c con 0x7f3ea80b4360 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:01.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eafdcb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea809dec0 con 0x7f3ea809ac40 2024-03-29T11:35:01.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eafdcb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea809da30 con 0x7f3ea80b4360 2024-03-29T11:35:01.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eafdcb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea80b39c0 con 0x7f3ea809c870 2024-03-29T11:35:01.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eafdcb640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eafdcb640 10 monclient(hunting): authenticate will time out at 867.618232s 2024-03-29T11:35:01.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eae368640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f3ea80b4360 0x7f3ea814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:39110/0 (socket says 172.21.15.146:39110) 2024-03-29T11:35:01.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eae368640 1 -- 172.21.15.146:0/2696420047 learned_addr learned my addr 172.21.15.146:0/2696420047 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:01.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 1 -- 172.21.15.146:0/2696420047 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218855527 0 0) 0x7f3ea809da30 con 0x7f3ea80b4360 2024-03-29T11:35:01.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:35:01.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 1 -- 172.21.15.146:0/2696420047 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e94003520 con 0x7f3ea80b4360 2024-03-29T11:35:01.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 1 -- 172.21.15.146:0/2696420047 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838393387 0 0) 0x7f3e98001110 con 0x7f3ea809ac40 2024-03-29T11:35:01.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:35:01.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 1 -- 172.21.15.146:0/2696420047 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ea809da30 con 0x7f3ea809ac40 2024-03-29T11:35:01.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 1 -- 172.21.15.146:0/2696420047 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679362310 0 0) 0x7f3ea80b39c0 con 0x7f3ea809c870 2024-03-29T11:35:01.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:35:01.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 1 -- 172.21.15.146:0/2696420047 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e98001110 con 0x7f3ea809c870 2024-03-29T11:35:01.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 1 -- 172.21.15.146:0/2696420047 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3514564553 0 0) 0x7f3e94003520 con 0x7f3ea80b4360 2024-03-29T11:35:01.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 1 -- 172.21.15.146:0/2696420047 >> v1:172.21.15.37:6790/0 conn(0x7f3ea809c870 legacy=0x7f3ea813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 1 -- 172.21.15.146:0/2696420047 >> v1:172.21.15.37:6789/0 conn(0x7f3ea809ac40 legacy=0x7f3ea80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 1 monclient: found mon.noname-c 2024-03-29T11:35:01.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 1 -- 172.21.15.146:0/2696420047 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea80b5160 con 0x7f3ea80b4360 2024-03-29T11:35:01.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 1 -- 172.21.15.146:0/2696420047 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3ea0001a10 con 0x7f3ea80b4360 2024-03-29T11:35:01.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eafdcb640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eafdcb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.374+0000 7f3eacb65640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eacb65640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eacb65640 10 monclient: dump: 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eafdcb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eacb65640 1 -- 172.21.15.146:0/2696420047 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ea0001bf0 con 0x7f3ea80b4360 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eacb65640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eafdcb640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eacb65640 1 -- 172.21.15.146:0/2696420047 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3ea0002000 con 0x7f3ea80b4360 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eafdcb640 4 set_mon_vals no callback set 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eacb65640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eacb65640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eacb65640 10 monclient: dump: 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.384 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eafdcb640 10 monclient: shutdown 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eafdcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eafdcb640 1 -- 172.21.15.146:0/2696420047 >> v1:172.21.15.146:6789/0 conn(0x7f3ea80b4360 legacy=0x7f3ea814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eafdcb640 1 -- 172.21.15.146:0/2696420047 shutdown_connections 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eafdcb640 1 -- 172.21.15.146:0/2696420047 >> 172.21.15.146:0/2696420047 conn(0x7f3ea8066b00 msgr2=0x7f3ea8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eafdcb640 1 -- 172.21.15.146:0/2696420047 shutdown_connections 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.375+0000 7f3eafdcb640 1 -- 172.21.15.146:0/2696420047 wait complete. 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.376+0000 7f3eafdcb640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.376+0000 7f3eafdcb640 10 monclient: monmap: 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:01.385 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-29T11:35:01.386 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-29T11:35:01.386 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.386 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.386 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:01.386 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:01.386 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:01.386 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.376+0000 7f3eafdcb640 1 librados: starting msgr at 2024-03-29T11:35:01.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.376+0000 7f3eafdcb640 1 librados: starting objecter 2024-03-29T11:35:01.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.376+0000 7f3eafdcb640 1 Processor -- start 2024-03-29T11:35:01.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.376+0000 7f3eafdcb640 1 -- start start 2024-03-29T11:35:01.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.376+0000 7f3eafdcb640 1 librados: setting wanted keys 2024-03-29T11:35:01.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.376+0000 7f3eafdcb640 1 librados: calling monclient init 2024-03-29T11:35:01.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.376+0000 7f3eafdcb640 10 monclient: init 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3eafdcb640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3eafdcb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3eafdcb640 10 monclient(hunting): picked mon.noname-a con 0x7f3ea809ac40 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3eafdcb640 10 monclient(hunting): picked mon.noname-b con 0x7f3ea8142c20 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3eafdcb640 10 monclient(hunting): picked mon.noname-c con 0x7f3ea81d7dc0 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3eafdcb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea81d4d30 con 0x7f3ea809ac40 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3eafdcb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea81430b0 con 0x7f3ea8142c20 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3eafdcb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea81d71b0 con 0x7f3ea81d7dc0 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3eafdcb640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3eafdcb640 10 monclient(hunting): authenticate will time out at 867.621101s 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3ead366640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f3ea8142c20 0x7f3ea81d76b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:39112/0 (socket says 172.21.15.146:39112) 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3ead366640 1 -- 172.21.15.146:0/3882880499 learned_addr learned my addr 172.21.15.146:0/3882880499 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3eadb67640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f3ea809ac40 0x7f3ea8143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.146:42564/0 (socket says 172.21.15.146:42564) 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 client.?.objecter ms_handle_connect 0x7f3ea8142c20 2024-03-29T11:35:01.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 client.?.objecter ms_handle_connect 0x7f3ea809ac40 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 client.?.objecter ms_handle_connect 0x7f3ea81d7dc0 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404959700 0 0) 0x7f3ea81430b0 con 0x7f3ea8142c20 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e8c004eb0 con 0x7f3ea8142c20 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101598037 0 0) 0x7f3ea81d4d30 con 0x7f3ea809ac40 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:35:01.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ea81430b0 con 0x7f3ea809ac40 2024-03-29T11:35:01.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042240339 0 0) 0x7f3ea81d71b0 con 0x7f3ea81d7dc0 2024-03-29T11:35:01.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:35:01.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ea81d4d30 con 0x7f3ea81d7dc0 2024-03-29T11:35:01.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1123949431 0 0) 0x7f3e8c004eb0 con 0x7f3ea8142c20 2024-03-29T11:35:01.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ea81d71b0 con 0x7f3ea8142c20 2024-03-29T11:35:01.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3ea4001a10 con 0x7f3ea8142c20 2024-03-29T11:35:01.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:01.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): dump: 2024-03-29T11:35:01.389 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 799884783 0 0) 0x7f3ea81430b0 con 0x7f3ea809ac40 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e8c004eb0 con 0x7f3ea809ac40 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3e980018a0 con 0x7f3ea809ac40 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:01.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): dump: 2024-03-29T11:35:01.391 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.391 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.391 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.391 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.391 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.391 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.391 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.391 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.391 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.391 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 50864411 0 0) 0x7f3ea81d4d30 con 0x7f3ea81d7dc0 2024-03-29T11:35:01.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ea81430b0 con 0x7f3ea81d7dc0 2024-03-29T11:35:01.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3ea0001f00 con 0x7f3ea81d7dc0 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.377+0000 7f3e9e7fc640 10 monclient(hunting): dump: 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 273535443 0 0) 0x7f3ea81d71b0 con 0x7f3ea8142c20 2024-03-29T11:35:01.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 >> v1:172.21.15.37:6790/0 conn(0x7f3ea81d7dc0 legacy=0x7f3ea8145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 >> v1:172.21.15.37:6789/0 conn(0x7f3ea809ac40 legacy=0x7f3ea8143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 1 monclient: found mon.b 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea8144ce0 con 0x7f3ea8142c20 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3eafdcb640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3eafdcb640 10 monclient: _renew_subs 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3eafdcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3eafdcb640 1 -- 172.21.15.146:0/3882880499 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea8145490 con 0x7f3ea8142c20 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3eafdcb640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:01.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3eafdcb640 10 monclient: _renew_subs 2024-03-29T11:35:01.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3eafdcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3eafdcb640 1 -- 172.21.15.146:0/3882880499 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea8146330 con 0x7f3ea8142c20 2024-03-29T11:35:01.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3eafdcb640 1 librados: init done 2024-03-29T11:35:01.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ea4001c70 con 0x7f3ea8142c20 2024-03-29T11:35:01.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3ea4003a60 con 0x7f3ea8142c20 2024-03-29T11:35:01.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 10 monclient: dump: 2024-03-29T11:35:01.394 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.394 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.394 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.394 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.394 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9ffff640 4 set_mon_vals no callback set 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3ea4004010 con 0x7f3ea8142c20 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1342363548 0 0) 0x7f3ea4006990 con 0x7f3ea8142c20 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 10 client.4153.objecter ms_dispatch 0x7f3ea8000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 3 client.4153.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 3 client.4153.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.378+0000 7f3e9e7fc640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.379+0000 7f3eafdcb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.379+0000 7f3eafdcb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.379+0000 7f3eafdcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.379+0000 7f3eafdcb640 1 -- 172.21.15.146:0/3882880499 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea809b020 con 0x7f3ea8142c20 2024-03-29T11:35:01.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.380+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3ea4004450 con 0x7f3ea8142c20 2024-03-29T11:35:01.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.381+0000 7f3e9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.381+0000 7f3e9e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:01.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.454+0000 7f3eafdcb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "634a1b82-a618-42c8-b306-b42760f2e055", "id": 6}] 2024-03-29T11:35:01.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.454+0000 7f3eafdcb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "634a1b82-a618-42c8-b306-b42760f2e055", "id": 6}] 2024-03-29T11:35:01.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.454+0000 7f3eafdcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.454+0000 7f3eafdcb640 1 -- 172.21.15.146:0/3882880499 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "634a1b82-a618-42c8-b306-b42760f2e055", "id": 6} v 0) v1 -- 0x7f3ea804f730 con 0x7f3ea8142c20 2024-03-29T11:35:01.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.458+0000 7f3e9e7fc640 1 -- 172.21.15.146:0/3882880499 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "634a1b82-a618-42c8-b306-b42760f2e055", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 399024557 0 1113811849) 0x7f3ea40063a0 con 0x7f3ea8142c20 2024-03-29T11:35:01.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.458+0000 7f3e9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "634a1b82-a618-42c8-b306-b42760f2e055", "id": 6}] 2024-03-29T11:35:01.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.458+0000 7f3e9e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:01.455 INFO:teuthology.orchestra.run.smithi146.stdout:6 2024-03-29T11:35:01.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.458+0000 7f3eafdcb640 10 librados: watch_flush enter 2024-03-29T11:35:01.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.459+0000 7f3eafdcb640 10 librados: watch_flush exit 2024-03-29T11:35:01.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.459+0000 7f3eafdcb640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T11:35:01.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.459+0000 7f3eafdcb640 10 client.4153.objecter successfully canceled tick 2024-03-29T11:35:01.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.459+0000 7f3eafdcb640 10 monclient: shutdown 2024-03-29T11:35:01.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.459+0000 7f3eafdcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.459+0000 7f3eafdcb640 1 -- 172.21.15.146:0/3882880499 >> v1:172.21.15.146:6789/0 conn(0x7f3ea8142c20 legacy=0x7f3ea81d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.459+0000 7f3eafdcb640 1 -- 172.21.15.146:0/3882880499 shutdown_connections 2024-03-29T11:35:01.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.459+0000 7f3eafdcb640 1 -- 172.21.15.146:0/3882880499 >> 172.21.15.146:0/3882880499 conn(0x7f3ea8066b00 msgr2=0x7f3ea813a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.459+0000 7f3eafdcb640 1 -- 172.21.15.146:0/3882880499 shutdown_connections 2024-03-29T11:35:01.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.459+0000 7f3eafdcb640 1 -- 172.21.15.146:0/3882880499 wait complete. 2024-03-29T11:35:01.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.459+0000 7f3eafdcb640 1 librados: shutdown 2024-03-29T11:35:01.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 35f0fe13-712e-4137-b072-6f0db1b9789b 7 2024-03-29T11:35:01.496 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:01.498+0000 7f935f24b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:35:01.529 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:01.531+0000 7f04d6813040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:35:01.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.569+0000 7fb8a86b3640 10 monclient: get_monmap_and_config 2024-03-29T11:35:01.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.569+0000 7fb8a86b3640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.569+0000 7fb8a86b3640 10 monclient: monmap: 2024-03-29T11:35:01.567 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:01.567 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.567 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:35:01.570629+0000 2024-03-29T11:35:01.568 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:35:01.570629+0000 2024-03-29T11:35:01.568 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.568 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.568 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:01.568 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:01.568 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:01.568 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.570+0000 7fb8a86b3640 1 Processor -- start 2024-03-29T11:35:01.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.570+0000 7fb8a86b3640 1 -- start start 2024-03-29T11:35:01.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.570+0000 7fb8a86b3640 10 monclient: init 2024-03-29T11:35:01.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a86b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a86b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:01.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a86b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb8a009ac40 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:01.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a86b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb8a009c870 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:01.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a86b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb8a00b4360 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a86b3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a009dec0 con 0x7fb8a00b4360 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a86b3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a009da30 con 0x7fb8a009ac40 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a86b3640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a00b39c0 con 0x7fb8a009c870 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a86b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a86b3640 10 monclient(hunting): authenticate will time out at 867.815161s 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a644f640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb8a009ac40 0x7fb8a00b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:39114/0 (socket says 172.21.15.146:39114) 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a644f640 1 -- 172.21.15.146:0/852065984 learned_addr learned my addr 172.21.15.146:0/852065984 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 1 -- 172.21.15.146:0/852065984 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927740198 0 0) 0x7fb8a009da30 con 0x7fb8a009ac40 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient(hunting): my global_id is 4156 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 1 -- 172.21.15.146:0/852065984 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb88c003520 con 0x7fb8a009ac40 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 1 -- 172.21.15.146:0/852065984 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464305534 0 0) 0x7fb898001490 con 0x7fb8a00b4360 2024-03-29T11:35:01.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 1 -- 172.21.15.146:0/852065984 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a009da30 con 0x7fb8a00b4360 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 1 -- 172.21.15.146:0/852065984 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793231105 0 0) 0x7fb8a00b39c0 con 0x7fb8a009c870 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 1 -- 172.21.15.146:0/852065984 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb898001490 con 0x7fb8a009c870 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 1 -- 172.21.15.146:0/852065984 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 814028388 0 0) 0x7fb88c003520 con 0x7fb8a009ac40 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 1 -- 172.21.15.146:0/852065984 >> v1:172.21.15.37:6790/0 conn(0x7fb8a009c870 legacy=0x7fb8a013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 1 -- 172.21.15.146:0/852065984 >> v1:172.21.15.37:6789/0 conn(0x7fb8a00b4360 legacy=0x7fb8a014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 1 monclient: found mon.noname-c 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 1 -- 172.21.15.146:0/852065984 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a00b5160 con 0x7fb8a009ac40 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a86b3640 5 monclient: authenticate success, global_id 4156 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 1 -- 172.21.15.146:0/852065984 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fb890001690 con 0x7fb8a009ac40 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a544d640 10 monclient: dump: 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.571+0000 7fb8a86b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a544d640 1 -- 172.21.15.146:0/852065984 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8900017f0 con 0x7fb8a009ac40 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a544d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a544d640 1 -- 172.21.15.146:0/852065984 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fb890001c00 con 0x7fb8a009ac40 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a86b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a86b3640 4 set_mon_vals no callback set 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a544d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a544d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a544d640 10 monclient: dump: 2024-03-29T11:35:01.580 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a86b3640 10 monclient: shutdown 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a86b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a86b3640 1 -- 172.21.15.146:0/852065984 >> v1:172.21.15.146:6789/0 conn(0x7fb8a009ac40 legacy=0x7fb8a00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a86b3640 1 -- 172.21.15.146:0/852065984 shutdown_connections 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a86b3640 1 -- 172.21.15.146:0/852065984 >> 172.21.15.146:0/852065984 conn(0x7fb8a0066b00 msgr2=0x7fb8a0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a86b3640 1 -- 172.21.15.146:0/852065984 shutdown_connections 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.572+0000 7fb8a86b3640 1 -- 172.21.15.146:0/852065984 wait complete. 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.573+0000 7fb8a86b3640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.573+0000 7fb8a86b3640 10 monclient: monmap: 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.573+0000 7fb8a86b3640 1 librados: starting msgr at 2024-03-29T11:35:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.573+0000 7fb8a86b3640 1 librados: starting objecter 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.573+0000 7fb8a86b3640 1 Processor -- start 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.573+0000 7fb8a86b3640 1 -- start start 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.573+0000 7fb8a86b3640 1 librados: setting wanted keys 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.573+0000 7fb8a86b3640 1 librados: calling monclient init 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.573+0000 7fb8a86b3640 10 monclient: init 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb8a86b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb8a86b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb8a86b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb8a009ac40 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb8a86b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb8a00afbb0 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb8a86b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb8a01db3a0 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb8a86b3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a00b15e0 con 0x7fb8a009ac40 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb8a86b3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a01da7e0 con 0x7fb8a00afbb0 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb8a86b3640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a00b1850 con 0x7fb8a01db3a0 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb8a86b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb8a86b3640 10 monclient(hunting): authenticate will time out at 867.818245s 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb8a5c4e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb8a00afbb0 0x7fb8a01dac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:39126/0 (socket says 172.21.15.146:39126) 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb8a5c4e640 1 -- 172.21.15.146:0/2372823533 learned_addr learned my addr 172.21.15.146:0/2372823533 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 client.?.objecter ms_handle_connect 0x7fb8a00afbb0 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 client.?.objecter ms_handle_connect 0x7fb8a009ac40 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558711481 0 0) 0x7fb8a01da7e0 con 0x7fb8a00afbb0 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): my global_id is 4159 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb884004eb0 con 0x7fb8a00afbb0 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 client.?.objecter ms_handle_connect 0x7fb8a01db3a0 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60475354 0 0) 0x7fb8a00b15e0 con 0x7fb8a009ac40 2024-03-29T11:35:01.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a01da7e0 con 0x7fb8a009ac40 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877257155 0 0) 0x7fb8a00b1850 con 0x7fb8a01db3a0 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a00b15e0 con 0x7fb8a01db3a0 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1924266510 0 0) 0x7fb884004eb0 con 0x7fb8a00afbb0 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8a00b1850 con 0x7fb8a00afbb0 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fb89c001930 con 0x7fb8a00afbb0 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.574+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2890817921 0 0) 0x7fb8a01da7e0 con 0x7fb8a009ac40 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb884004eb0 con 0x7fb8a009ac40 2024-03-29T11:35:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fb890001b00 con 0x7fb8a009ac40 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1046918333 0 0) 0x7fb8a00b1850 con 0x7fb8a00afbb0 2024-03-29T11:35:01.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 >> v1:172.21.15.37:6790/0 conn(0x7fb8a01db3a0 legacy=0x7fb8a0138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 >> v1:172.21.15.37:6789/0 conn(0x7fb8a009ac40 legacy=0x7fb8a00b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 1 monclient: found mon.b 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a00b0a90 con 0x7fb8a00afbb0 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb8a86b3640 5 monclient: authenticate success, global_id 4159 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb8a86b3640 10 monclient: _renew_subs 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb8a86b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb8a86b3640 1 -- 172.21.15.146:0/2372823533 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a00acd00 con 0x7fb8a00afbb0 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb8a86b3640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:01.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb8a86b3640 10 monclient: _renew_subs 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb8a86b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb8a86b3640 1 -- 172.21.15.146:0/2372823533 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a01398e0 con 0x7fb8a00afbb0 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb8a86b3640 1 librados: init done 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb89c003610 con 0x7fb8a00afbb0 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fb89c0039e0 con 0x7fb8a00afbb0 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb8a4c4c640 4 set_mon_vals no callback set 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 10 monclient: dump: 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb89c003f90 con 0x7fb8a00afbb0 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 235179854 0 0) 0x7fb89c0068b0 con 0x7fb8a00afbb0 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 10 client.4159.objecter ms_dispatch 0x7fb8a0000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 3 client.4159.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 3 client.4159.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.575+0000 7fb896ffd640 20 client.4159.objecter dump_active .. 0 homeless 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.576+0000 7fb8a86b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.576+0000 7fb8a86b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.576+0000 7fb8a86b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.576+0000 7fb8a86b3640 1 -- 172.21.15.146:0/2372823533 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb888005080 con 0x7fb8a00afbb0 2024-03-29T11:35:01.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.578+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb89c0043d0 con 0x7fb8a00afbb0 2024-03-29T11:35:01.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.578+0000 7fb896ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.578+0000 7fb896ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:01.636 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:01.638+0000 7f935f24b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.650+0000 7fb8a86b3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "35f0fe13-712e-4137-b072-6f0db1b9789b", "id": 7}] 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.650+0000 7fb8a86b3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "35f0fe13-712e-4137-b072-6f0db1b9789b", "id": 7}] 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.650+0000 7fb8a86b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.650+0000 7fb8a86b3640 1 -- 172.21.15.146:0/2372823533 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "35f0fe13-712e-4137-b072-6f0db1b9789b", "id": 7} v 0) v1 -- 0x7fb8880058b0 con 0x7fb8a00afbb0 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.654+0000 7fb896ffd640 1 -- 172.21.15.146:0/2372823533 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "35f0fe13-712e-4137-b072-6f0db1b9789b", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1767489108 0 1371665406) 0x7fb89c006ac0 con 0x7fb8a00afbb0 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.654+0000 7fb896ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "35f0fe13-712e-4137-b072-6f0db1b9789b", "id": 7}] 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.654+0000 7fb896ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi146.stdout:7 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.654+0000 7fb8a86b3640 10 librados: watch_flush enter 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.654+0000 7fb8a86b3640 10 librados: watch_flush exit 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.654+0000 7fb8a86b3640 20 client.4159.objecter shutdown clearing up homeless session... 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.654+0000 7fb8a86b3640 10 client.4159.objecter successfully canceled tick 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.654+0000 7fb8a86b3640 10 monclient: shutdown 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.654+0000 7fb8a86b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.654+0000 7fb8a86b3640 1 -- 172.21.15.146:0/2372823533 >> v1:172.21.15.146:6789/0 conn(0x7fb8a00afbb0 legacy=0x7fb8a01dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.655+0000 7fb8a86b3640 1 -- 172.21.15.146:0/2372823533 shutdown_connections 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.655+0000 7fb8a86b3640 1 -- 172.21.15.146:0/2372823533 >> 172.21.15.146:0/2372823533 conn(0x7fb8a0066b00 msgr2=0x7fb8a013c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.655+0000 7fb8a86b3640 1 -- 172.21.15.146:0/2372823533 shutdown_connections 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.655+0000 7fb8a86b3640 1 -- 172.21.15.146:0/2372823533 wait complete. 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-29T11:35:01.655+0000 7fb8a86b3640 1 librados: shutdown 2024-03-29T11:35:01.663 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:35:01.663 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:35:01.665 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:35:01.665 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:35:01.665 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:35:01.667 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:35:01.667 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:35:01.667 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:35:01.669 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:35:01.669 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:35:01.670 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:35:01.671 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:35:01.672 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:35:01.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:35:01.673 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:01.675+0000 7f04d6813040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:35:01.674 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:35:01.674 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:35:01.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:35:01.676 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:35:01.676 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:35:01.677 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:35:01.679 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:35:01.679 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:35:01.679 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:35:01.681 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:35:01.681 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:01.794 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:01.795+0000 7f935f24b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:35:01.807 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:01.809+0000 7f04d6813040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:35:01.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.816+0000 7f0678915640 10 monclient: get_monmap_and_config 2024-03-29T11:35:01.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.816+0000 7f0678915640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.816+0000 7f0678915640 10 monclient: monmap: 2024-03-29T11:35:01.815 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-03-29T11:35:01.815 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.816 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-29T11:35:01.818735+0000 2024-03-29T11:35:01.816 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-29T11:35:01.818735+0000 2024-03-29T11:35:01.816 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.816 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:01.816 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:01.816 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:01.816 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:01.816 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:35:01.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.817+0000 7f0678915640 1 Processor -- start 2024-03-29T11:35:01.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.817+0000 7f0678915640 1 -- start start 2024-03-29T11:35:01.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.817+0000 7f0678915640 10 monclient: init 2024-03-29T11:35:01.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f0678915640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f0678915640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:01.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f0678915640 10 monclient(hunting): picked mon.noname-b con 0x7f06740b8240 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:01.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f0678915640 10 monclient(hunting): picked mon.noname-c con 0x7f06740b1f60 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:01.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f0678915640 10 monclient(hunting): picked mon.noname-a con 0x7f067409c8c0 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:01.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f0678915640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067409bd80 con 0x7f067409c8c0 2024-03-29T11:35:01.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f0678915640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067409b960 con 0x7f06740b1f60 2024-03-29T11:35:01.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f0678915640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06740b2ec0 con 0x7f06740b8240 2024-03-29T11:35:01.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f0678915640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f0678915640 10 monclient(hunting): authenticate will time out at 881.268278s 2024-03-29T11:35:01.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f0672d9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f067409c8c0 0x7f067414d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41126/0 (socket says 172.21.15.37:41126) 2024-03-29T11:35:01.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067259c640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f06740b8240 0x7f067413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:43824/0 (socket says 172.21.15.37:43824) 2024-03-29T11:35:01.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f0672d9d640 1 -- 172.21.15.37:0/1595678222 learned_addr learned my addr 172.21.15.37:0/1595678222 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 1 -- 172.21.15.37:0/1595678222 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219314537 0 0) 0x7f06740b2ec0 con 0x7f06740b8240 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 1 -- 172.21.15.37:0/1595678222 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0660003520 con 0x7f06740b8240 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 1 -- 172.21.15.37:0/1595678222 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934043835 0 0) 0x7f067409bd80 con 0x7f067409c8c0 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 1 -- 172.21.15.37:0/1595678222 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06740b2ec0 con 0x7f067409c8c0 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 1 -- 172.21.15.37:0/1595678222 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006241138 0 0) 0x7f067409b960 con 0x7f06740b1f60 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.818+0000 7f067159a640 1 -- 172.21.15.37:0/1595678222 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f067409bd80 con 0x7f06740b1f60 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 1 -- 172.21.15.37:0/1595678222 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1623629705 0 0) 0x7f0660003520 con 0x7f06740b8240 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 1 -- 172.21.15.37:0/1595678222 >> v1:172.21.15.146:6789/0 conn(0x7f06740b1f60 legacy=0x7f067409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 1 -- 172.21.15.37:0/1595678222 >> v1:172.21.15.37:6789/0 conn(0x7f067409c8c0 legacy=0x7f067414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 1 monclient: found mon.noname-b 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 1 -- 172.21.15.37:0/1595678222 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067413fb60 con 0x7f06740b8240 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 1 -- 172.21.15.37:0/1595678222 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0664001690 con 0x7f06740b8240 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 10 monclient: dump: 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f0678915640 5 monclient: authenticate success, global_id 4181 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f0678915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 1 -- 172.21.15.37:0/1595678222 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0664001870 con 0x7f06740b8240 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f0678915640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f0678915640 4 set_mon_vals no callback set 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 1 -- 172.21.15.37:0/1595678222 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0664001c80 con 0x7f06740b8240 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f067159a640 10 monclient: dump: 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:01.830 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.831 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.831 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.831 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:35:01.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f0678915640 10 monclient: shutdown 2024-03-29T11:35:01.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f0678915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f0678915640 1 -- 172.21.15.37:0/1595678222 >> v1:172.21.15.37:6790/0 conn(0x7f06740b8240 legacy=0x7f067413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f0678915640 1 -- 172.21.15.37:0/1595678222 shutdown_connections 2024-03-29T11:35:01.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f0678915640 1 -- 172.21.15.37:0/1595678222 >> 172.21.15.37:0/1595678222 conn(0x7f0674066b00 msgr2=0x7f0674066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f0678915640 1 -- 172.21.15.37:0/1595678222 shutdown_connections 2024-03-29T11:35:01.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.819+0000 7f0678915640 1 -- 172.21.15.37:0/1595678222 wait complete. 2024-03-29T11:35:01.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.820+0000 7f0678915640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.820+0000 7f0678915640 10 monclient: monmap: 2024-03-29T11:35:01.831 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-03-29T11:35:01.831 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.820+0000 7f0678915640 1 librados: starting msgr at 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.820+0000 7f0678915640 1 librados: starting objecter 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 1 Processor -- start 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 1 -- start start 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 1 librados: setting wanted keys 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 1 librados: calling monclient init 2024-03-29T11:35:01.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 10 monclient: init 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 10 monclient(hunting): picked mon.noname-b con 0x7f067409c8c0 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 10 monclient(hunting): picked mon.noname-a con 0x7f06740b8240 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 10 monclient(hunting): picked mon.noname-c con 0x7f0674144ab0 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06741358e0 con 0x7f06740b8240 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0674143e60 con 0x7f067409c8c0 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06740b1b70 con 0x7f0674144ab0 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0671d9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f06740b8240 0x7f06741421e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41138/0 (socket says 172.21.15.37:41138) 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0671d9b640 1 -- 172.21.15.37:0/1069134478 learned_addr learned my addr 172.21.15.37:0/1069134478 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f0678915640 10 monclient(hunting): authenticate will time out at 881.271662s 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 10 client.?.objecter ms_handle_connect 0x7f06740b8240 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 10 client.?.objecter ms_handle_connect 0x7f0674144ab0 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361461757 0 0) 0x7f06740b1b70 con 0x7f0674144ab0 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0654004eb0 con 0x7f0674144ab0 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032227086 0 0) 0x7f06741358e0 con 0x7f06740b8240 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06740b1b70 con 0x7f06740b8240 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 10 client.?.objecter ms_handle_connect 0x7f067409c8c0 2024-03-29T11:35:01.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.821+0000 7f066affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323809271 0 0) 0x7f0674143e60 con 0x7f067409c8c0 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06741358e0 con 0x7f067409c8c0 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 9550635 0 0) 0x7f0654004eb0 con 0x7f0674144ab0 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0674143e60 con 0x7f0674144ab0 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f066c001ab0 con 0x7f0674144ab0 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient(hunting): dump: 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.835 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2035867649 0 0) 0x7f06740b1b70 con 0x7f06740b8240 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0654004eb0 con 0x7f06740b8240 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f065c0019b0 con 0x7f06740b8240 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient(hunting): dump: 2024-03-29T11:35:01.836 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4024053447 0 0) 0x7f06741358e0 con 0x7f067409c8c0 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06740b1b70 con 0x7f067409c8c0 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0664001b80 con 0x7f067409c8c0 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient(hunting): dump: 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 383375894 0 0) 0x7f0674143e60 con 0x7f0674144ab0 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 >> v1:172.21.15.146:6789/0 conn(0x7f067409c8c0 legacy=0x7f06740acee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 >> v1:172.21.15.37:6789/0 conn(0x7f06740b8240 legacy=0x7f06741421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 monclient: found mon.c 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0674146090 con 0x7f0674144ab0 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f0678915640 5 monclient: authenticate success, global_id 4184 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f0678915640 10 monclient: _renew_subs 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f0678915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f0678915640 1 -- 172.21.15.37:0/1069134478 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0674146370 con 0x7f0674144ab0 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f0678915640 10 client.4184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f0678915640 10 monclient: _renew_subs 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f0678915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f0678915640 1 -- 172.21.15.37:0/1069134478 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0674145140 con 0x7f0674144ab0 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f066c001fc0 con 0x7f0674144ab0 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f0678915640 1 librados: init done 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f066c004d00 con 0x7f0674144ab0 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 monclient: dump: 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f0670d99640 4 set_mon_vals no callback set 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f066c0051f0 con 0x7f0674144ab0 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f066c005b70 con 0x7f0674144ab0 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 10 client.4184.objecter ms_dispatch 0x7f0674000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 3 client.4184.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 3 client.4184.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.822+0000 7f066affd640 20 client.4184.objecter dump_active .. 0 homeless 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.823+0000 7f0678915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.823+0000 7f0678915640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.823+0000 7f0678915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.823+0000 7f0678915640 1 -- 172.21.15.37:0/1069134478 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0658005080 con 0x7f0674144ab0 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.825+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f066c005d30 con 0x7f0674144ab0 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.825+0000 7f066affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.825+0000 7f066affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:01.843 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:01.845+0000 7f935f24b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:35:01.853 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:01.856+0000 7f04d6813040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:35:01.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.887+0000 7f0678915640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:01.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.887+0000 7f0678915640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:01.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.887+0000 7f0678915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:01.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.887+0000 7f0678915640 1 -- 172.21.15.37:0/1069134478 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0658005330 con 0x7f0674144ab0 2024-03-29T11:35:01.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.887+0000 7f066affd640 1 -- 172.21.15.37:0/1069134478 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f066c0054a0 con 0x7f0674144ab0 2024-03-29T11:35:01.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.887+0000 7f066affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:01.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.887+0000 7f066affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:01.887 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-29T11:35:01.887 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:01.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.888+0000 7f0678915640 10 librados: watch_flush enter 2024-03-29T11:35:01.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.888+0000 7f0678915640 10 librados: watch_flush exit 2024-03-29T11:35:01.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.888+0000 7f0678915640 20 client.4184.objecter shutdown clearing up homeless session... 2024-03-29T11:35:01.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.888+0000 7f0678915640 10 client.4184.objecter successfully canceled tick 2024-03-29T11:35:01.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.888+0000 7f0678915640 10 monclient: shutdown 2024-03-29T11:35:01.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.888+0000 7f0678915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.888+0000 7f0678915640 1 -- 172.21.15.37:0/1069134478 >> v1:172.21.15.37:6790/0 conn(0x7f0674144ab0 legacy=0x7f06741e1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.888+0000 7f0678915640 1 -- 172.21.15.37:0/1069134478 shutdown_connections 2024-03-29T11:35:01.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.888+0000 7f0678915640 1 -- 172.21.15.37:0/1069134478 >> 172.21.15.37:0/1069134478 conn(0x7f0674066b00 msgr2=0x7f067406bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.888+0000 7f0678915640 1 -- 172.21.15.37:0/1069134478 shutdown_connections 2024-03-29T11:35:01.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.889+0000 7f0678915640 1 -- 172.21.15.37:0/1069134478 wait complete. 2024-03-29T11:35:01.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-29T11:35:01.889+0000 7f0678915640 1 librados: shutdown 2024-03-29T11:35:01.898 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:35:01.898 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:35:01.962 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:01.963+0000 7f935f24b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:35:01.965 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:01.968+0000 7f04d6813040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:35:02.060 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:02.062+0000 7f935f24b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:35:02.063 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:02.066+0000 7f04d6813040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:35:02.218 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:02.220+0000 7f04d6813040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:35:02.220 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:02.222+0000 7f935f24b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:35:02.260 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:02.263+0000 7f04d6813040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:35:02.264 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:02.265+0000 7f935f24b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:35:02.265 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-03-29T11:35:02.266+0000 7fe6aaa2a640 -1 Falling back to public interface 2024-03-29T11:35:02.274 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-03-29T11:35:02.275+0000 7ff58db9e640 -1 Falling back to public interface 2024-03-29T11:35:02.280 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-03-29T11:35:02.283+0000 7f66c3889640 -1 Falling back to public interface 2024-03-29T11:35:02.290 INFO:tasks.ceph.osd.3.smithi037.stderr:2024-03-29T11:35:02.291+0000 7ff28ae11640 -1 Falling back to public interface 2024-03-29T11:35:02.291 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-03-29T11:35:02.294+0000 7f1693ad3640 -1 Falling back to public interface 2024-03-29T11:35:02.298 INFO:tasks.ceph.osd.2.smithi037.stderr:2024-03-29T11:35:02.299+0000 7f3e78413640 -1 Falling back to public interface 2024-03-29T11:35:02.300 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-03-29T11:35:02.303+0000 7fc8f3b1f640 -1 Falling back to public interface 2024-03-29T11:35:02.301 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-03-29T11:35:02.304+0000 7f8aa4c0c640 -1 Falling back to public interface 2024-03-29T11:35:02.419 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:02.422+0000 7f04d6813040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:35:02.425 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:02.427+0000 7f935f24b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:35:02.704 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:02.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.795+0000 7f91b6855640 10 monclient: get_monmap_and_config 2024-03-29T11:35:02.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.795+0000 7f91b6855640 10 monclient: build_initial_monmap 2024-03-29T11:35:02.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.795+0000 7f91b6855640 10 monclient: monmap: 2024-03-29T11:35:02.794 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:02.794 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:02.794 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:35:02.797439+0000 2024-03-29T11:35:02.795 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:35:02.797439+0000 2024-03-29T11:35:02.795 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:02.795 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:02.795 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:02.795 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:02.795 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:02.795 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:02.802 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.796+0000 7f91b6855640 1 Processor -- start 2024-03-29T11:35:02.802 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.796+0000 7f91b6855640 1 -- start start 2024-03-29T11:35:02.802 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.796+0000 7f91b6855640 10 monclient: init 2024-03-29T11:35:02.802 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.796+0000 7f91b6855640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.796+0000 7f91b6855640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.796+0000 7f91b6855640 10 monclient(hunting): picked mon.noname-c con 0x7f91b0070170 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.796+0000 7f91b6855640 10 monclient(hunting): picked mon.noname-a con 0x7f91b00713f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.796+0000 7f91b6855640 10 monclient(hunting): picked mon.noname-b con 0x7f91b0070550 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.796+0000 7f91b6855640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b013dbd0 con 0x7f91b00713f0 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.796+0000 7f91b6855640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b013ff70 con 0x7f91b0070170 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.796+0000 7f91b6855640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b0140250 con 0x7f91b0070550 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.796+0000 7f91b6855640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.796+0000 7f91b6855640 10 monclient(hunting): authenticate will time out at 882.246921s 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91af7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f91b00713f0 0x7f91b01406e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41152/0 (socket says 172.21.15.37:41152) 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91af7fe640 1 -- 172.21.15.37:0/3714959655 learned_addr learned my addr 172.21.15.37:0/3714959655 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 1 -- 172.21.15.37:0/3714959655 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611229139 0 0) 0x7f91a4001110 con 0x7f91b00713f0 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 1 -- 172.21.15.37:0/3714959655 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9198003520 con 0x7f91b00713f0 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 1 -- 172.21.15.37:0/3714959655 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975896786 0 0) 0x7f91b0140250 con 0x7f91b0070550 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 1 -- 172.21.15.37:0/3714959655 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a4001110 con 0x7f91b0070550 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 1 -- 172.21.15.37:0/3714959655 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501148869 0 0) 0x7f91b013ff70 con 0x7f91b0070170 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient(hunting): my global_id is 4192 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 1 -- 172.21.15.37:0/3714959655 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b0140250 con 0x7f91b0070170 2024-03-29T11:35:02.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 1 -- 172.21.15.37:0/3714959655 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1114130919 0 0) 0x7f9198003520 con 0x7f91b00713f0 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 1 -- 172.21.15.37:0/3714959655 >> v1:172.21.15.37:6790/0 conn(0x7f91b0070550 legacy=0x7f91b014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 1 -- 172.21.15.37:0/3714959655 >> v1:172.21.15.146:6789/0 conn(0x7f91b0070170 legacy=0x7f91b013d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 1 monclient: found mon.noname-a 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 1 -- 172.21.15.37:0/3714959655 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b0140df0 con 0x7f91b00713f0 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91b6855640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 1 -- 172.21.15.37:0/3714959655 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f91a4001680 con 0x7f91b00713f0 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91b6855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:02.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient: dump: 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91b6855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 1 -- 172.21.15.37:0/3714959655 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91a4001860 con 0x7f91b00713f0 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 1 -- 172.21.15.37:0/3714959655 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f91a4001c70 con 0x7f91b00713f0 2024-03-29T11:35:02.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91b6855640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91b6855640 4 set_mon_vals no callback set 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.797+0000 7f91aeffd640 10 monclient: dump: 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.798+0000 7f91b6855640 10 monclient: shutdown 2024-03-29T11:35:02.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.798+0000 7f91b6855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.798+0000 7f91b6855640 1 -- 172.21.15.37:0/3714959655 >> v1:172.21.15.37:6789/0 conn(0x7f91b00713f0 legacy=0x7f91b01406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.798+0000 7f91b6855640 1 -- 172.21.15.37:0/3714959655 shutdown_connections 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.798+0000 7f91b6855640 1 -- 172.21.15.37:0/3714959655 >> 172.21.15.37:0/3714959655 conn(0x7f91b0067610 msgr2=0x7f91b00b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.798+0000 7f91b6855640 1 -- 172.21.15.37:0/3714959655 shutdown_connections 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.798+0000 7f91b6855640 1 -- 172.21.15.37:0/3714959655 wait complete. 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.799+0000 7f91b6855640 10 monclient: build_initial_monmap 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.799+0000 7f91b6855640 10 monclient: monmap: 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:02.808 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.799+0000 7f91b6855640 1 librados: starting msgr at 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.799+0000 7f91b6855640 1 librados: starting objecter 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.799+0000 7f91b6855640 1 Processor -- start 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.799+0000 7f91b6855640 1 -- start start 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.799+0000 7f91b6855640 1 librados: setting wanted keys 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.799+0000 7f91b6855640 1 librados: calling monclient init 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.799+0000 7f91b6855640 10 monclient: init 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91b6855640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91b6855640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91b6855640 10 monclient(hunting): picked mon.noname-c con 0x7f91b0070170 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91b6855640 10 monclient(hunting): picked mon.noname-a con 0x7f91b0070550 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91b6855640 10 monclient(hunting): picked mon.noname-b con 0x7f91b00713f0 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91b6855640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b00b4f80 con 0x7f91b0070550 2024-03-29T11:35:02.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91b6855640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b01425e0 con 0x7f91b00713f0 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91b6855640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b0133570 con 0x7f91b0070170 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91b6855640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91b6855640 10 monclient(hunting): authenticate will time out at 882.250166s 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91affff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f91b0070170 0x7f91b01d70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:43852/0 (socket says 172.21.15.37:43852) 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91affff640 1 -- 172.21.15.37:0/2175362073 learned_addr learned my addr 172.21.15.37:0/2175362073 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91af7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f91b0070550 0x7f91b0132750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41162/0 (socket says 172.21.15.37:41162) 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 client.?.objecter ms_handle_connect 0x7f91b0070550 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 client.?.objecter ms_handle_connect 0x7f91b0070170 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655679934 0 0) 0x7f91b0133570 con 0x7f91b0070170 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f919c004ee0 con 0x7f91b0070170 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 client.?.objecter ms_handle_connect 0x7f91b00713f0 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342049190 0 0) 0x7f91b00b4f80 con 0x7f91b0070550 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b0133570 con 0x7f91b0070550 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233879943 0 0) 0x7f91b01425e0 con 0x7f91b00713f0 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): my global_id is 4195 2024-03-29T11:35:02.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b00b4f80 con 0x7f91b00713f0 2024-03-29T11:35:02.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3263771807 0 0) 0x7f91b0133570 con 0x7f91b0070550 2024-03-29T11:35:02.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91b01425e0 con 0x7f91b0070550 2024-03-29T11:35:02.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1528183341 0 0) 0x7f919c004ee0 con 0x7f91b0070170 2024-03-29T11:35:02.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91b0133570 con 0x7f91b0070170 2024-03-29T11:35:02.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f91a40012a0 con 0x7f91b0070550 2024-03-29T11:35:02.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:02.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): dump: 2024-03-29T11:35:02.812 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:02.812 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:02.812 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:02.812 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:02.812 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.813 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:02.813 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:02.813 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:02.813 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:02.813 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:02.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:02.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f91a00018f0 con 0x7f91b0070170 2024-03-29T11:35:02.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:02.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): dump: 2024-03-29T11:35:02.813 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:02.813 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:02.813 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:02.813 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:02.814 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.814 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:02.814 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:02.814 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:02.814 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:02.814 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:02.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1915450062 0 0) 0x7f91b00b4f80 con 0x7f91b00713f0 2024-03-29T11:35:02.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f919c004ee0 con 0x7f91b00713f0 2024-03-29T11:35:02.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f91a80013a0 con 0x7f91b00713f0 2024-03-29T11:35:02.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:02.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.800+0000 7f91acff9640 10 monclient(hunting): dump: 2024-03-29T11:35:02.814 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 382110335 0 0) 0x7f91b01425e0 con 0x7f91b0070550 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 >> v1:172.21.15.37:6790/0 conn(0x7f91b0070170 legacy=0x7f91b01d70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 >> v1:172.21.15.146:6789/0 conn(0x7f91b00713f0 legacy=0x7f91b0132e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 1 monclient: found mon.a 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:02.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b01d7800 con 0x7f91b0070550 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91b6855640 5 monclient: authenticate success, global_id 4194 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91b6855640 10 monclient: _renew_subs 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91b6855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91b6855640 1 -- 172.21.15.37:0/2175362073 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b00b4a60 con 0x7f91b0070550 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91b6855640 10 client.4194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91b6855640 10 monclient: _renew_subs 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91b6855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91b6855640 1 -- 172.21.15.37:0/2175362073 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b00b55f0 con 0x7f91b0070550 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91b6855640 1 librados: init done 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91a4004c60 con 0x7f91b0070550 2024-03-29T11:35:02.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f91a4005030 con 0x7f91b0070550 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91ae7fc640 4 set_mon_vals no callback set 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 10 monclient: dump: 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:02.817 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f91a40055e0 con 0x7f91b0070550 2024-03-29T11:35:02.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f91a4005f60 con 0x7f91b0070550 2024-03-29T11:35:02.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 10 client.4194.objecter ms_dispatch 0x7f91b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:02.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 3 client.4194.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:02.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 3 client.4194.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:02.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.801+0000 7f91acff9640 20 client.4194.objecter dump_active .. 0 homeless 2024-03-29T11:35:02.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.802+0000 7f91b6855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.802+0000 7f91b6855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.802+0000 7f91b6855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:02.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.802+0000 7f91b6855640 1 -- 172.21.15.37:0/2175362073 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9190005080 con 0x7f91b0070550 2024-03-29T11:35:02.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.804+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f91a4006260 con 0x7f91b0070550 2024-03-29T11:35:02.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.804+0000 7f91acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.804+0000 7f91acff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:02.865 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.866+0000 7f91b6855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:02.866 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.867+0000 7f91b6855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:02.866 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.867+0000 7f91b6855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:02.866 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.867+0000 7f91b6855640 1 -- 172.21.15.37:0/2175362073 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9190005330 con 0x7f91b0070550 2024-03-29T11:35:02.866 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.867+0000 7f91acff9640 1 -- 172.21.15.37:0/2175362073 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f91a4005890 con 0x7f91b0070550 2024-03-29T11:35:02.866 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.867+0000 7f91acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:02.866 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.867+0000 7f91acff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:02.866 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:35:02.867 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:02.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.868+0000 7f91b6855640 10 librados: watch_flush enter 2024-03-29T11:35:02.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.868+0000 7f91b6855640 10 librados: watch_flush exit 2024-03-29T11:35:02.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.868+0000 7f91b6855640 20 client.4194.objecter shutdown clearing up homeless session... 2024-03-29T11:35:02.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.868+0000 7f91b6855640 10 client.4194.objecter successfully canceled tick 2024-03-29T11:35:02.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.868+0000 7f91b6855640 10 monclient: shutdown 2024-03-29T11:35:02.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.868+0000 7f91b6855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:02.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.868+0000 7f91b6855640 1 -- 172.21.15.37:0/2175362073 >> v1:172.21.15.37:6789/0 conn(0x7f91b0070550 legacy=0x7f91b0132750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.868+0000 7f91b6855640 1 -- 172.21.15.37:0/2175362073 shutdown_connections 2024-03-29T11:35:02.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.868+0000 7f91b6855640 1 -- 172.21.15.37:0/2175362073 >> 172.21.15.37:0/2175362073 conn(0x7f91b0067610 msgr2=0x7f91b0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:02.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.868+0000 7f91b6855640 1 -- 172.21.15.37:0/2175362073 shutdown_connections 2024-03-29T11:35:02.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.868+0000 7f91b6855640 1 -- 172.21.15.37:0/2175362073 wait complete. 2024-03-29T11:35:02.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:02.868+0000 7f91b6855640 1 librados: shutdown 2024-03-29T11:35:02.876 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:03.251 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:03.253+0000 7f04d6813040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:35:03.259 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:03.261+0000 7f935f24b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:35:03.528 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:03.530+0000 7f04d6813040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:35:03.545 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:03.546+0000 7f935f24b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:35:03.579 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:03.581+0000 7f04d6813040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:35:03.598 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:03.599+0000 7f935f24b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:35:03.696 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:03.699+0000 7f04d6813040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:35:03.720 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-29T11:35:03.722+0000 7f935f24b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:35:04.711 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:04.713+0000 7f0477170640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:35:04.711 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-29T11:35:04.714+0000 7f0477170640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:35:05.729 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-03-29T11:35:05.730+0000 7fe6aaa2a640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:35:05.788 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-03-29T11:35:05.791+0000 7f66c3889640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:35:05.795 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-03-29T11:35:05.797+0000 7ff58db9e640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:35:05.799 INFO:tasks.ceph.osd.2.smithi037.stderr:2024-03-29T11:35:05.800+0000 7f3e78413640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:35:05.800 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-03-29T11:35:05.803+0000 7f1693ad3640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:35:05.803 INFO:tasks.ceph.osd.3.smithi037.stderr:2024-03-29T11:35:05.804+0000 7ff28ae11640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:35:05.838 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-03-29T11:35:05.841+0000 7fc8f3b1f640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:35:05.839 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-03-29T11:35:05.842+0000 7f8aa4c0c640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:35:09.682 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:09.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.828+0000 7ff33e715640 10 monclient: get_monmap_and_config 2024-03-29T11:35:09.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.828+0000 7ff33e715640 10 monclient: build_initial_monmap 2024-03-29T11:35:09.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.828+0000 7ff33e715640 10 monclient: monmap: 2024-03-29T11:35:09.828 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:09.828 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:09.828 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:35:09.830681+0000 2024-03-29T11:35:09.828 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:35:09.830681+0000 2024-03-29T11:35:09.828 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:09.828 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:09.829 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:09.829 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:09.829 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:09.829 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:09.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.830+0000 7ff33e715640 1 Processor -- start 2024-03-29T11:35:09.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.830+0000 7ff33e715640 1 -- start start 2024-03-29T11:35:09.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.830+0000 7ff33e715640 10 monclient: init 2024-03-29T11:35:09.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.831+0000 7ff33e715640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:09.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.831+0000 7ff33e715640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:09.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.831+0000 7ff33e715640 10 monclient(hunting): picked mon.noname-a con 0x7ff3380708a0 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:09.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.831+0000 7ff33e715640 10 monclient(hunting): picked mon.noname-c con 0x7ff33806fe90 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:09.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.831+0000 7ff33e715640 10 monclient(hunting): picked mon.noname-b con 0x7ff338143c50 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:09.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.831+0000 7ff32ffff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff33806fe90 0x7ff338143540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:37052/0 (socket says 172.21.15.37:37052) 2024-03-29T11:35:09.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.831+0000 7ff32ffff640 1 -- 172.21.15.37:0/1418520219 learned_addr learned my addr 172.21.15.37:0/1418520219 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:09.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.831+0000 7ff337fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff3380708a0 0x7ff3380af390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:45726/0 (socket says 172.21.15.37:45726) 2024-03-29T11:35:09.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.831+0000 7ff33e715640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff338072630 con 0x7ff3380708a0 2024-03-29T11:35:09.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.832+0000 7ff33e715640 1 -- 172.21.15.37:0/1418520219 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff338071410 con 0x7ff33806fe90 2024-03-29T11:35:09.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.832+0000 7ff33e715640 1 -- 172.21.15.37:0/1418520219 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3380afaa0 con 0x7ff338143c50 2024-03-29T11:35:09.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.832+0000 7ff33e715640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:09.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.832+0000 7ff33e715640 10 monclient(hunting): authenticate will time out at 889.282047s 2024-03-29T11:35:09.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.832+0000 7ff3377fe640 1 -- 172.21.15.37:0/1418520219 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322184540 0 0) 0x7ff338071410 con 0x7ff33806fe90 2024-03-29T11:35:09.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.832+0000 7ff3377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:09.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.832+0000 7ff3377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:09.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.832+0000 7ff3377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:09.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.832+0000 7ff3377fe640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:35:09.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.832+0000 7ff3377fe640 1 -- 172.21.15.37:0/1418520219 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff324003520 con 0x7ff33806fe90 2024-03-29T11:35:09.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 1 -- 172.21.15.37:0/1418520219 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555429021 0 0) 0x7ff3380afaa0 con 0x7ff338143c50 2024-03-29T11:35:09.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:09.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:09.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:09.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient(hunting): my global_id is 4232 2024-03-29T11:35:09.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 1 -- 172.21.15.37:0/1418520219 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff324003840 con 0x7ff338143c50 2024-03-29T11:35:09.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 1 -- 172.21.15.37:0/1418520219 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568681347 0 0) 0x7ff338072630 con 0x7ff3380708a0 2024-03-29T11:35:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:35:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 1 -- 172.21.15.37:0/1418520219 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3380afaa0 con 0x7ff3380708a0 2024-03-29T11:35:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 1 -- 172.21.15.37:0/1418520219 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 980675927 0 0) 0x7ff324003520 con 0x7ff33806fe90 2024-03-29T11:35:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 1 -- 172.21.15.37:0/1418520219 >> v1:172.21.15.37:6790/0 conn(0x7ff338143c50 legacy=0x7ff33814b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 1 -- 172.21.15.37:0/1418520219 >> v1:172.21.15.37:6789/0 conn(0x7ff3380708a0 legacy=0x7ff3380af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 1 monclient: found mon.noname-c 2024-03-29T11:35:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:35:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 1 -- 172.21.15.37:0/1418520219 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3380ac8b0 con 0x7ff33806fe90 2024-03-29T11:35:09.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:09.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:09.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:09.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 1 -- 172.21.15.37:0/1418520219 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7ff3280019f0 con 0x7ff33806fe90 2024-03-29T11:35:09.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:09.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:09.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff3377fe640 10 monclient: dump: 2024-03-29T11:35:09.836 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:09.836 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:09.837 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:09.837 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:09.837 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:09.837 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:09.837 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:09.837 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:09.837 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:09.837 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:09.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff33e715640 5 monclient: authenticate success, global_id 4237 2024-03-29T11:35:09.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.833+0000 7ff33e715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:09.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.834+0000 7ff3377fe640 1 -- 172.21.15.37:0/1418520219 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff328001bd0 con 0x7ff33806fe90 2024-03-29T11:35:09.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.834+0000 7ff3377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:09.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.834+0000 7ff3377fe640 1 -- 172.21.15.37:0/1418520219 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7ff328001fe0 con 0x7ff33806fe90 2024-03-29T11:35:09.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.834+0000 7ff3377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:09.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.834+0000 7ff3377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:09.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.834+0000 7ff3377fe640 10 monclient: dump: 2024-03-29T11:35:09.845 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:09.845 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:09.845 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:09.845 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:09.845 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:09.845 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:09.845 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:09.846 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:09.846 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:09.846 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:09.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.834+0000 7ff33e715640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:09.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.834+0000 7ff33e715640 4 set_mon_vals no callback set 2024-03-29T11:35:09.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.835+0000 7ff33e715640 10 monclient: shutdown 2024-03-29T11:35:09.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.835+0000 7ff33e715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:09.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.835+0000 7ff33e715640 1 -- 172.21.15.37:0/1418520219 >> v1:172.21.15.146:6789/0 conn(0x7ff33806fe90 legacy=0x7ff338143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:09.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.835+0000 7ff33e715640 1 -- 172.21.15.37:0/1418520219 shutdown_connections 2024-03-29T11:35:09.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.835+0000 7ff33e715640 1 -- 172.21.15.37:0/1418520219 >> 172.21.15.37:0/1418520219 conn(0x7ff3380675b0 msgr2=0x7ff3380b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:09.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.835+0000 7ff33e715640 1 -- 172.21.15.37:0/1418520219 shutdown_connections 2024-03-29T11:35:09.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.835+0000 7ff33e715640 1 -- 172.21.15.37:0/1418520219 wait complete. 2024-03-29T11:35:09.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.836+0000 7ff33e715640 10 monclient: build_initial_monmap 2024-03-29T11:35:09.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.836+0000 7ff33e715640 10 monclient: monmap: 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.836+0000 7ff33e715640 1 librados: starting msgr at 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.836+0000 7ff33e715640 1 librados: starting objecter 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.836+0000 7ff33e715640 1 Processor -- start 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.836+0000 7ff33e715640 1 -- start start 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.836+0000 7ff33e715640 1 librados: setting wanted keys 2024-03-29T11:35:09.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.836+0000 7ff33e715640 1 librados: calling monclient init 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.836+0000 7ff33e715640 10 monclient: init 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.837+0000 7ff33e715640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.837+0000 7ff33e715640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.837+0000 7ff33e715640 10 monclient(hunting): picked mon.noname-c con 0x7ff33806fe90 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.837+0000 7ff33e715640 10 monclient(hunting): picked mon.noname-b con 0x7ff3380708a0 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.837+0000 7ff33e715640 10 monclient(hunting): picked mon.noname-a con 0x7ff338143c50 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff33e715640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3381d22c0 con 0x7ff338143c50 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff32ffff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff3380708a0 0x7ff3381dc0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:37172/0 (socket says 172.21.15.37:37172) 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff32ffff640 1 -- 172.21.15.37:0/633067254 learned_addr learned my addr 172.21.15.37:0/633067254 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff33e715640 1 -- 172.21.15.37:0/633067254 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3381dbbb0 con 0x7ff3380708a0 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff33e715640 1 -- 172.21.15.37:0/633067254 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3380ac5b0 con 0x7ff33806fe90 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff33e715640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff33e715640 10 monclient(hunting): authenticate will time out at 889.288051s 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 10 client.?.objecter ms_handle_connect 0x7ff3380708a0 2024-03-29T11:35:09.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:09.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 10 client.?.objecter ms_handle_connect 0x7ff338143c50 2024-03-29T11:35:09.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:09.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305652051 0 0) 0x7ff3381dbbb0 con 0x7ff3380708a0 2024-03-29T11:35:09.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:09.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:09.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:09.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:35:09.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff318004eb0 con 0x7ff3380708a0 2024-03-29T11:35:09.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 10 client.?.objecter ms_handle_connect 0x7ff33806fe90 2024-03-29T11:35:09.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:09.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240123576 0 0) 0x7ff3381d22c0 con 0x7ff338143c50 2024-03-29T11:35:09.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:09.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:09.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:09.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:35:09.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.838+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3381dbbb0 con 0x7ff338143c50 2024-03-29T11:35:09.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1444914852 0 0) 0x7ff318004eb0 con 0x7ff3380708a0 2024-03-29T11:35:09.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3381d22c0 con 0x7ff3380708a0 2024-03-29T11:35:09.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7ff328001fa0 con 0x7ff3380708a0 2024-03-29T11:35:09.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:09.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:09.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 10 monclient(hunting): dump: 2024-03-29T11:35:09.850 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:09.850 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:09.850 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:09.850 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:09.850 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:09.850 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607307688 0 0) 0x7ff3380ac5b0 con 0x7ff33806fe90 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff318004eb0 con 0x7ff33806fe90 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2915786414 0 0) 0x7ff3381dbbb0 con 0x7ff338143c50 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3380ac5b0 con 0x7ff338143c50 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7ff330001790 con 0x7ff338143c50 2024-03-29T11:35:09.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.839+0000 7ff3357fa640 10 monclient(hunting): dump: 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 156216570 0 0) 0x7ff3381d22c0 con 0x7ff3380708a0 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 >> v1:172.21.15.37:6790/0 conn(0x7ff33806fe90 legacy=0x7ff3380abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:09.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 >> v1:172.21.15.37:6789/0 conn(0x7ff338143c50 legacy=0x7ff3380a9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 10 monclient: _finish_hunting 0 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 1 monclient: found mon.b 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3381dc7c0 con 0x7ff3380708a0 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 10 monclient: _finish_auth 0 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 10 monclient: _check_auth_tickets 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff33e715640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff33e715640 10 monclient: _renew_subs 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff33e715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff33e715640 1 -- 172.21.15.37:0/633067254 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3380a94e0 con 0x7ff3380708a0 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff33e715640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff33e715640 10 monclient: _renew_subs 2024-03-29T11:35:09.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff33e715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff33e715640 1 -- 172.21.15.37:0/633067254 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff338140d70 con 0x7ff3380708a0 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff33e715640 1 librados: init done 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff328004e80 con 0x7ff3380708a0 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7ff328002100 con 0x7ff3380708a0 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff3357fa640 10 monclient: dump: 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:09.854 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:09.855 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:09.855 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:09.855 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:09.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.840+0000 7ff336ffd640 4 set_mon_vals no callback set 2024-03-29T11:35:09.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.842+0000 7ff33e715640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:09.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.843+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7ff328005290 con 0x7ff3380708a0 2024-03-29T11:35:09.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.843+0000 7ff33e715640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:09.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.843+0000 7ff33e715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:09.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.843+0000 7ff33e715640 1 -- 172.21.15.37:0/633067254 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff338071e40 con 0x7ff3380708a0 2024-03-29T11:35:09.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.844+0000 7ff337fff640 10 monclient: get_auth_request con 0x7ff31806df90 auth_method 0 2024-03-29T11:35:09.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.844+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7ff328084f10 con 0x7ff3380708a0 2024-03-29T11:35:09.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.844+0000 7ff3357fa640 10 client.4240.objecter ms_dispatch 0x7ff338000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:09.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.844+0000 7ff3357fa640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:09.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.844+0000 7ff3357fa640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:09.856 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.844+0000 7ff3357fa640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-29T11:35:09.856 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.844+0000 7ff3357fa640 10 client.4240.objecter ms_handle_connect 0x7ff31806df90 2024-03-29T11:35:09.856 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.851+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3280507f0 con 0x7ff3380708a0 2024-03-29T11:35:09.856 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.851+0000 7ff3357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:09.856 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:09.851+0000 7ff3357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:10.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.227+0000 7ff33e715640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:10.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.227+0000 7ff33e715640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:10.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.227+0000 7ff33e715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:10.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.227+0000 7ff33e715640 1 -- 172.21.15.37:0/633067254 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3381dc950 con 0x7ff3380708a0 2024-03-29T11:35:10.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.228+0000 7ff3357fa640 1 -- 172.21.15.37:0/633067254 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7ff3381dc950 con 0x7ff3380708a0 2024-03-29T11:35:10.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.228+0000 7ff3357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:10.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.228+0000 7ff3357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:10.227 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:35:10.228 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:10.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.232+0000 7ff33e715640 10 librados: watch_flush enter 2024-03-29T11:35:10.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.232+0000 7ff33e715640 10 librados: watch_flush exit 2024-03-29T11:35:10.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.232+0000 7ff33e715640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-29T11:35:10.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.232+0000 7ff33e715640 10 client.4240.objecter successfully canceled tick 2024-03-29T11:35:10.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.233+0000 7ff33e715640 1 -- 172.21.15.37:0/633067254 >> v1:172.21.15.146:6816/2899149139 conn(0x7ff31806df90 legacy=0x7ff318070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.233+0000 7ff33e715640 10 monclient: shutdown 2024-03-29T11:35:10.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.233+0000 7ff33e715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:10.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.233+0000 7ff33e715640 1 -- 172.21.15.37:0/633067254 >> v1:172.21.15.146:6789/0 conn(0x7ff3380708a0 legacy=0x7ff3381dc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.233+0000 7ff33e715640 1 -- 172.21.15.37:0/633067254 shutdown_connections 2024-03-29T11:35:10.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.233+0000 7ff33e715640 1 -- 172.21.15.37:0/633067254 >> 172.21.15.37:0/633067254 conn(0x7ff3380675b0 msgr2=0x7ff3380b0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:10.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.233+0000 7ff33e715640 1 -- 172.21.15.37:0/633067254 shutdown_connections 2024-03-29T11:35:10.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.233+0000 7ff33e715640 1 -- 172.21.15.37:0/633067254 wait complete. 2024-03-29T11:35:10.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:10.233+0000 7ff33e715640 1 librados: shutdown 2024-03-29T11:35:10.247 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:17.054 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:17.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.212+0000 7f0715796640 10 monclient: get_monmap_and_config 2024-03-29T11:35:17.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.212+0000 7f0715796640 10 monclient: build_initial_monmap 2024-03-29T11:35:17.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.212+0000 7f0715796640 10 monclient: monmap: 2024-03-29T11:35:17.211 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:17.211 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:17.211 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:35:17.213479+0000 2024-03-29T11:35:17.211 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:35:17.213479+0000 2024-03-29T11:35:17.211 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:17.211 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:17.211 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:17.211 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:17.211 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:17.211 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:17.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.213+0000 7f0715796640 1 Processor -- start 2024-03-29T11:35:17.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.213+0000 7f0715796640 1 -- start start 2024-03-29T11:35:17.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.213+0000 7f0715796640 10 monclient: init 2024-03-29T11:35:17.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.214+0000 7f0715796640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:17.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.214+0000 7f0715796640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:17.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.214+0000 7f0715796640 10 monclient(hunting): picked mon.noname-c con 0x7f07101457f0 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:17.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.214+0000 7f0715796640 10 monclient(hunting): picked mon.noname-a con 0x7f071013f650 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:17.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.214+0000 7f0715796640 10 monclient(hunting): picked mon.noname-b con 0x7f07100a9d70 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:17.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.214+0000 7f0715796640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07100aaef0 con 0x7f071013f650 2024-03-29T11:35:17.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.214+0000 7f0715796640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0710145bd0 con 0x7f07101457f0 2024-03-29T11:35:17.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.214+0000 7f0715796640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0710145eb0 con 0x7f07100a9d70 2024-03-29T11:35:17.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.214+0000 7f0715796640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:17.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.214+0000 7f0715796640 10 monclient(hunting): authenticate will time out at 896.663483s 2024-03-29T11:35:17.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.214+0000 7f070effd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f07101457f0 0x7f071013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:38106/0 (socket says 172.21.15.37:38106) 2024-03-29T11:35:17.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.214+0000 7f070effd640 1 -- 172.21.15.37:0/1953503416 learned_addr learned my addr 172.21.15.37:0/1953503416 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:17.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.214+0000 7f070dffb640 1 -- 172.21.15.37:0/1953503416 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164823296 0 0) 0x7f0710145bd0 con 0x7f07101457f0 2024-03-29T11:35:17.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:35:17.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 1 -- 172.21.15.37:0/1953503416 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f8003520 con 0x7f07101457f0 2024-03-29T11:35:17.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 1 -- 172.21.15.37:0/1953503416 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4211412042 0 0) 0x7f06f8003520 con 0x7f07101457f0 2024-03-29T11:35:17.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 1 -- 172.21.15.37:0/1953503416 >> v1:172.21.15.37:6790/0 conn(0x7f07100a9d70 legacy=0x7f071014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 1 -- 172.21.15.37:0/1953503416 >> v1:172.21.15.37:6789/0 conn(0x7f071013f650 legacy=0x7f07100b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:17.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:17.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 1 monclient: found mon.noname-c 2024-03-29T11:35:17.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:17.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:35:17.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 1 -- 172.21.15.37:0/1953503416 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0710146340 con 0x7f07101457f0 2024-03-29T11:35:17.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:17.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:17.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:17.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 1 -- 172.21.15.37:0/1953503416 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0700001690 con 0x7f07101457f0 2024-03-29T11:35:17.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:17.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient: dump: 2024-03-29T11:35:17.216 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f0715796640 5 monclient: authenticate success, global_id 4243 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f0715796640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 1 -- 172.21.15.37:0/1953503416 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0700001920 con 0x7f07101457f0 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 1 -- 172.21.15.37:0/1953503416 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0700001690 con 0x7f07101457f0 2024-03-29T11:35:17.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f070dffb640 10 monclient: dump: 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f0715796640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.215+0000 7f0715796640 4 set_mon_vals no callback set 2024-03-29T11:35:17.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.216+0000 7f0715796640 10 monclient: shutdown 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.216+0000 7f0715796640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.216+0000 7f0715796640 1 -- 172.21.15.37:0/1953503416 >> v1:172.21.15.146:6789/0 conn(0x7f07101457f0 legacy=0x7f071013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.216+0000 7f0715796640 1 -- 172.21.15.37:0/1953503416 shutdown_connections 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.216+0000 7f0715796640 1 -- 172.21.15.37:0/1953503416 >> 172.21.15.37:0/1953503416 conn(0x7f0710066a00 msgr2=0x7f0710066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.216+0000 7f0715796640 1 -- 172.21.15.37:0/1953503416 shutdown_connections 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.216+0000 7f0715796640 1 -- 172.21.15.37:0/1953503416 wait complete. 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.217+0000 7f0715796640 10 monclient: build_initial_monmap 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.217+0000 7f0715796640 10 monclient: monmap: 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:17.219 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:17.220 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:17.220 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:17.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.217+0000 7f0715796640 1 librados: starting msgr at 2024-03-29T11:35:17.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.217+0000 7f0715796640 1 librados: starting objecter 2024-03-29T11:35:17.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.218+0000 7f0715796640 1 Processor -- start 2024-03-29T11:35:17.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.218+0000 7f0715796640 1 -- start start 2024-03-29T11:35:17.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.218+0000 7f0715796640 1 librados: setting wanted keys 2024-03-29T11:35:17.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.218+0000 7f0715796640 1 librados: calling monclient init 2024-03-29T11:35:17.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.218+0000 7f0715796640 10 monclient: init 2024-03-29T11:35:17.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f0715796640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:17.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f0715796640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:17.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f0715796640 10 monclient(hunting): picked mon.noname-c con 0x7f07100a9d70 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:17.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f0715796640 10 monclient(hunting): picked mon.noname-b con 0x7f071013f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:17.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f0715796640 10 monclient(hunting): picked mon.noname-a con 0x7f07101457f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:17.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f070effd640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f07100a9d70 0x7f07101413f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:48286/0 (socket says 172.21.15.37:48286) 2024-03-29T11:35:17.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f070effd640 1 -- 172.21.15.37:0/1911009627 learned_addr learned my addr 172.21.15.37:0/1911009627 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:17.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f0715796640 1 -- 172.21.15.37:0/1911009627 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f071013b340 con 0x7f07101457f0 2024-03-29T11:35:17.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f0715796640 1 -- 172.21.15.37:0/1911009627 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f071013ae00 con 0x7f071013f650 2024-03-29T11:35:17.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f06f77fe640 10 client.?.objecter ms_handle_connect 0x7f07101457f0 2024-03-29T11:35:17.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f06f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:17.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f06f77fe640 10 client.?.objecter ms_handle_connect 0x7f071013f650 2024-03-29T11:35:17.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f06f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:17.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f0715796640 1 -- 172.21.15.37:0/1911009627 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f071013b590 con 0x7f07100a9d70 2024-03-29T11:35:17.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f06f77fe640 10 client.?.objecter ms_handle_connect 0x7f07100a9d70 2024-03-29T11:35:17.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f06f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:17.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.219+0000 7f0715796640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:17.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f0715796640 10 monclient(hunting): authenticate will time out at 896.668969s 2024-03-29T11:35:17.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385602321 0 0) 0x7f071013b590 con 0x7f07100a9d70 2024-03-29T11:35:17.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:35:17.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06fc004eb0 con 0x7f07100a9d70 2024-03-29T11:35:17.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970249008 0 0) 0x7f071013b340 con 0x7f07101457f0 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f071013b590 con 0x7f07101457f0 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431132623 0 0) 0x7f071013ae00 con 0x7f071013f650 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.220+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f071013b340 con 0x7f071013f650 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3145865167 0 0) 0x7f071013b590 con 0x7f07101457f0 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06fc006680 con 0x7f07101457f0 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0708001bb0 con 0x7f07101457f0 2024-03-29T11:35:17.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:17.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 10 monclient(hunting): dump: 2024-03-29T11:35:17.226 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:17.226 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:17.226 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:17.226 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:17.226 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.226 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:17.226 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:17.226 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:17.226 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:17.226 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:17.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:17.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1225975438 0 0) 0x7f06fc004eb0 con 0x7f07100a9d70 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f071013b590 con 0x7f07100a9d70 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0700001690 con 0x7f07100a9d70 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 10 monclient(hunting): dump: 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:17.227 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2798008630 0 0) 0x7f071013b340 con 0x7f071013f650 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06fc004eb0 con 0x7f071013f650 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0704001490 con 0x7f071013f650 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 10 monclient(hunting): dump: 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:17.228 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 50307544 0 0) 0x7f06fc006680 con 0x7f07101457f0 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 >> v1:172.21.15.37:6790/0 conn(0x7f07100a9d70 legacy=0x7f07101413f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 >> v1:172.21.15.146:6789/0 conn(0x7f071013f650 legacy=0x7f0710137450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 1 monclient: found mon.a 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071013b870 con 0x7f07101457f0 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.221+0000 7f06f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f0715796640 5 monclient: authenticate success, global_id 4245 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f0715796640 10 monclient: _renew_subs 2024-03-29T11:35:17.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f0715796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:17.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f0715796640 1 -- 172.21.15.37:0/1911009627 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0710141b00 con 0x7f07101457f0 2024-03-29T11:35:17.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f0715796640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:17.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f0715796640 10 monclient: _renew_subs 2024-03-29T11:35:17.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f0715796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:17.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f0715796640 1 -- 172.21.15.37:0/1911009627 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07101dfa20 con 0x7f07101457f0 2024-03-29T11:35:17.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f0715796640 1 librados: init done 2024-03-29T11:35:17.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07080020a0 con 0x7f07101457f0 2024-03-29T11:35:17.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f06f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:17.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f070d7fa640 4 set_mon_vals no callback set 2024-03-29T11:35:17.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0708003b10 con 0x7f07101457f0 2024-03-29T11:35:17.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f06f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f06f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:17.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.222+0000 7f06f77fe640 10 monclient: dump: 2024-03-29T11:35:17.230 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:17.231 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:17.231 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:17.231 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:17.231 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.231 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:17.231 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:17.231 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:17.231 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:17.231 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:17.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.223+0000 7f0715796640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:17.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.223+0000 7f0715796640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:17.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.223+0000 7f0715796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:17.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.223+0000 7f0715796640 1 -- 172.21.15.37:0/1911009627 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f0005080 con 0x7f07101457f0 2024-03-29T11:35:17.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.224+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f0708003c70 con 0x7f07101457f0 2024-03-29T11:35:17.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.225+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f0708083a50 con 0x7f07101457f0 2024-03-29T11:35:17.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.225+0000 7f06f77fe640 10 client.4245.objecter ms_dispatch 0x7f0710000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:17.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.225+0000 7f06f77fe640 3 client.4245.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:17.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.225+0000 7f06f77fe640 3 client.4245.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:17.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.225+0000 7f06f77fe640 20 client.4245.objecter dump_active .. 0 homeless 2024-03-29T11:35:17.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.225+0000 7f070effd640 10 monclient: get_auth_request con 0x7f06fc06e3b0 auth_method 0 2024-03-29T11:35:17.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.226+0000 7f06f77fe640 10 client.4245.objecter ms_handle_connect 0x7f06fc06e3b0 2024-03-29T11:35:17.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.234+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f070804f160 con 0x7f07101457f0 2024-03-29T11:35:17.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.234+0000 7f06f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:17.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.234+0000 7f06f77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:17.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.599+0000 7f0715796640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:17.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.599+0000 7f0715796640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:17.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.599+0000 7f0715796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:17.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.599+0000 7f0715796640 1 -- 172.21.15.37:0/1911009627 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06f0005330 con 0x7f07101457f0 2024-03-29T11:35:17.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.600+0000 7f06f77fe640 1 -- 172.21.15.37:0/1911009627 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f0708003970 con 0x7f07101457f0 2024-03-29T11:35:17.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.600+0000 7f06f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:17.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.600+0000 7f06f77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:17.599 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:35:17.599 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:17.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.603+0000 7f0715796640 10 librados: watch_flush enter 2024-03-29T11:35:17.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.603+0000 7f0715796640 10 librados: watch_flush exit 2024-03-29T11:35:17.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.603+0000 7f0715796640 20 client.4245.objecter shutdown clearing up homeless session... 2024-03-29T11:35:17.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.603+0000 7f0715796640 10 client.4245.objecter successfully canceled tick 2024-03-29T11:35:17.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.603+0000 7f0715796640 1 -- 172.21.15.37:0/1911009627 >> v1:172.21.15.146:6816/2899149139 conn(0x7f06fc06e3b0 legacy=0x7f06fc070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.603+0000 7f0715796640 10 monclient: shutdown 2024-03-29T11:35:17.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.603+0000 7f0715796640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:17.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.603+0000 7f0715796640 1 -- 172.21.15.37:0/1911009627 >> v1:172.21.15.37:6789/0 conn(0x7f07101457f0 legacy=0x7f07101e6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.604+0000 7f0715796640 1 -- 172.21.15.37:0/1911009627 shutdown_connections 2024-03-29T11:35:17.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.604+0000 7f0715796640 1 -- 172.21.15.37:0/1911009627 >> 172.21.15.37:0/1911009627 conn(0x7f0710066a00 msgr2=0x7f07100b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:17.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.604+0000 7f0715796640 1 -- 172.21.15.37:0/1911009627 shutdown_connections 2024-03-29T11:35:17.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.604+0000 7f0715796640 1 -- 172.21.15.37:0/1911009627 wait complete. 2024-03-29T11:35:17.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:17.604+0000 7f0715796640 1 librados: shutdown 2024-03-29T11:35:17.617 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:24.424 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:24.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.559+0000 7f5934b0e640 10 monclient: get_monmap_and_config 2024-03-29T11:35:24.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.559+0000 7f5934b0e640 10 monclient: build_initial_monmap 2024-03-29T11:35:24.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.559+0000 7f5934b0e640 10 monclient: monmap: 2024-03-29T11:35:24.557 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:24.557 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:24.557 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:35:24.560045+0000 2024-03-29T11:35:24.557 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:35:24.560045+0000 2024-03-29T11:35:24.557 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:24.558 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:24.558 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:24.558 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:24.558 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:24.558 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:24.558 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.559+0000 7f5934b0e640 1 Processor -- start 2024-03-29T11:35:24.559 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.559+0000 7f5934b0e640 1 -- start start 2024-03-29T11:35:24.559 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.559+0000 7f5934b0e640 10 monclient: init 2024-03-29T11:35:24.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.560+0000 7f5934b0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:24.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.560+0000 7f5934b0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:24.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f5934b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f59301457f0 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:24.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f5934b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f593013f650 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:24.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f5934b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f59300a9d70 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:24.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f5934b0e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59300aaef0 con 0x7f593013f650 2024-03-29T11:35:24.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f5934b0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5930145bd0 con 0x7f59301457f0 2024-03-29T11:35:24.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f5934b0e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5930145eb0 con 0x7f59300a9d70 2024-03-29T11:35:24.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f592ed9d640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f59300a9d70 0x7f593014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:33378/0 (socket says 172.21.15.37:33378) 2024-03-29T11:35:24.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f592ed9d640 1 -- 172.21.15.37:0/2161812074 learned_addr learned my addr 172.21.15.37:0/2161812074 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:24.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f5934b0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:24.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f5934b0e640 10 monclient(hunting): authenticate will time out at 904.010262s 2024-03-29T11:35:24.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f592d59a640 1 -- 172.21.15.37:0/2161812074 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901460130 0 0) 0x7f5930145eb0 con 0x7f59300a9d70 2024-03-29T11:35:24.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f592d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:24.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f592d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:24.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f592d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:24.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f592d59a640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:35:24.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.561+0000 7f592d59a640 1 -- 172.21.15.37:0/2161812074 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5918003520 con 0x7f59300a9d70 2024-03-29T11:35:24.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.562+0000 7f592d59a640 1 -- 172.21.15.37:0/2161812074 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3443245234 0 0) 0x7f5918003520 con 0x7f59300a9d70 2024-03-29T11:35:24.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.562+0000 7f592d59a640 1 -- 172.21.15.37:0/2161812074 >> v1:172.21.15.146:6789/0 conn(0x7f59301457f0 legacy=0x7f593013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:24.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.562+0000 7f592d59a640 1 -- 172.21.15.37:0/2161812074 >> v1:172.21.15.37:6789/0 conn(0x7f593013f650 legacy=0x7f59300b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:24.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:35:24.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 1 monclient: found mon.noname-b 2024-03-29T11:35:24.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:24.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:35:24.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 1 -- 172.21.15.37:0/2161812074 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5930146340 con 0x7f59300a9d70 2024-03-29T11:35:24.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 10 monclient: _finish_auth 0 2024-03-29T11:35:24.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:35:24.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:24.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 1 -- 172.21.15.37:0/2161812074 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5928001a10 con 0x7f59300a9d70 2024-03-29T11:35:24.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:24.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:24.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 10 monclient: dump: 2024-03-29T11:35:24.565 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:24.565 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:24.566 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:24.566 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:24.566 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:24.566 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:24.566 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:24.566 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:24.566 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:24.566 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:24.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f5934b0e640 5 monclient: authenticate success, global_id 4241 2024-03-29T11:35:24.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f5934b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:24.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 1 -- 172.21.15.37:0/2161812074 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5928001be0 con 0x7f59300a9d70 2024-03-29T11:35:24.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:24.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 1 -- 172.21.15.37:0/2161812074 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5928002030 con 0x7f59300a9d70 2024-03-29T11:35:24.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:24.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:24.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.564+0000 7f592d59a640 10 monclient: dump: 2024-03-29T11:35:24.567 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:24.567 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:24.567 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:24.567 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:24.567 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:24.567 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:24.567 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:24.568 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:24.568 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:24.568 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:24.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.565+0000 7f5934b0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:24.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.565+0000 7f5934b0e640 4 set_mon_vals no callback set 2024-03-29T11:35:24.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.565+0000 7f5934b0e640 10 monclient: shutdown 2024-03-29T11:35:24.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.566+0000 7f5934b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:24.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.566+0000 7f5934b0e640 1 -- 172.21.15.37:0/2161812074 >> v1:172.21.15.37:6790/0 conn(0x7f59300a9d70 legacy=0x7f593014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:24.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.566+0000 7f5934b0e640 1 -- 172.21.15.37:0/2161812074 shutdown_connections 2024-03-29T11:35:24.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.566+0000 7f5934b0e640 1 -- 172.21.15.37:0/2161812074 >> 172.21.15.37:0/2161812074 conn(0x7f5930066a00 msgr2=0x7f5930066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:24.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.566+0000 7f5934b0e640 1 -- 172.21.15.37:0/2161812074 shutdown_connections 2024-03-29T11:35:24.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.566+0000 7f5934b0e640 1 -- 172.21.15.37:0/2161812074 wait complete. 2024-03-29T11:35:24.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.567+0000 7f5934b0e640 10 monclient: build_initial_monmap 2024-03-29T11:35:24.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.567+0000 7f5934b0e640 10 monclient: monmap: 2024-03-29T11:35:24.575 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:24.575 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:24.575 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.568+0000 7f5934b0e640 1 librados: starting msgr at 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.568+0000 7f5934b0e640 1 librados: starting objecter 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.568+0000 7f5934b0e640 1 Processor -- start 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.568+0000 7f5934b0e640 1 -- start start 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.568+0000 7f5934b0e640 1 librados: setting wanted keys 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.568+0000 7f5934b0e640 1 librados: calling monclient init 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.568+0000 7f5934b0e640 10 monclient: init 2024-03-29T11:35:24.576 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.569+0000 7f5934b0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:24.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.569+0000 7f5934b0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:24.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.570+0000 7f5934b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f59300a9d70 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:24.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.570+0000 7f5934b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f593013f650 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:24.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.570+0000 7f5934b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f59301457f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:24.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.570+0000 7f5934b0e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593013db50 con 0x7f59301457f0 2024-03-29T11:35:24.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.570+0000 7f5934b0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593013cd40 con 0x7f59300a9d70 2024-03-29T11:35:24.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.570+0000 7f5934b0e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59301416a0 con 0x7f593013f650 2024-03-29T11:35:24.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.570+0000 7f5934b0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:24.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.570+0000 7f5934b0e640 10 monclient(hunting): authenticate will time out at 904.019490s 2024-03-29T11:35:24.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.570+0000 7f592e59c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f59300a9d70 0x7f59301358d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:59742/0 (socket says 172.21.15.37:59742) 2024-03-29T11:35:24.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.570+0000 7f592e59c640 1 -- 172.21.15.37:0/357021486 learned_addr learned my addr 172.21.15.37:0/357021486 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:24.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 10 client.?.objecter ms_handle_connect 0x7f593013f650 2024-03-29T11:35:24.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:24.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910435144 0 0) 0x7f59301416a0 con 0x7f593013f650 2024-03-29T11:35:24.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:24.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:24.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:24.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:35:24.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5910004eb0 con 0x7f593013f650 2024-03-29T11:35:24.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2982855173 0 0) 0x7f5910004eb0 con 0x7f593013f650 2024-03-29T11:35:24.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59301416a0 con 0x7f593013f650 2024-03-29T11:35:24.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5924001870 con 0x7f593013f650 2024-03-29T11:35:24.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 10 monclient(hunting): dump: 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1301361888 0 0) 0x7f59301416a0 con 0x7f593013f650 2024-03-29T11:35:24.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 >> v1:172.21.15.146:6789/0 conn(0x7f59300a9d70 legacy=0x7f59301358d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:24.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.571+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 >> v1:172.21.15.37:6789/0 conn(0x7f59301457f0 legacy=0x7f59301e1df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:24.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f591effd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:24.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f591effd640 1 monclient: found mon.c 2024-03-29T11:35:24.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f591effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:24.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f591effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:24.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59301402b0 con 0x7f593013f650 2024-03-29T11:35:24.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f591effd640 10 monclient: _finish_auth 0 2024-03-29T11:35:24.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f591effd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:24.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f591effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:24.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f5934b0e640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:35:24.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f5934b0e640 10 monclient: _renew_subs 2024-03-29T11:35:24.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f5934b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:24.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f5934b0e640 1 -- 172.21.15.37:0/357021486 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5930140510 con 0x7f593013f650 2024-03-29T11:35:24.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f5934b0e640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:24.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f5934b0e640 10 monclient: _renew_subs 2024-03-29T11:35:24.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f5934b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:24.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f5934b0e640 1 -- 172.21.15.37:0/357021486 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f593013e310 con 0x7f593013f650 2024-03-29T11:35:24.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.572+0000 7f5934b0e640 1 librados: init done 2024-03-29T11:35:24.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.574+0000 7f5934b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:24.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.574+0000 7f5934b0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:24.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.574+0000 7f5934b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:24.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.574+0000 7f5934b0e640 1 -- 172.21.15.37:0/357021486 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593013e9b0 con 0x7f593013f650 2024-03-29T11:35:24.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.574+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59240019d0 con 0x7f593013f650 2024-03-29T11:35:24.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.574+0000 7f591effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:24.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.574+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5924003850 con 0x7f593013f650 2024-03-29T11:35:24.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.574+0000 7f591effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:24.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.574+0000 7f591effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:24.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.574+0000 7f591effd640 10 monclient: dump: 2024-03-29T11:35:24.582 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:24.582 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:24.582 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:24.582 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:24.582 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:24.582 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:24.582 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:24.582 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:24.583 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:24.583 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:24.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.574+0000 7f592cd99640 4 set_mon_vals no callback set 2024-03-29T11:35:24.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.575+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f59240198e0 con 0x7f593013f650 2024-03-29T11:35:24.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.576+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f5924083900 con 0x7f593013f650 2024-03-29T11:35:24.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.576+0000 7f591effd640 10 client.4244.objecter ms_dispatch 0x7f5930000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:24.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.576+0000 7f591effd640 3 client.4244.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:24.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.576+0000 7f591effd640 3 client.4244.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:24.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.576+0000 7f591effd640 20 client.4244.objecter dump_active .. 0 homeless 2024-03-29T11:35:24.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.577+0000 7f592e59c640 10 monclient: get_auth_request con 0x7f591006db20 auth_method 0 2024-03-29T11:35:24.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.577+0000 7f591effd640 10 client.4244.objecter ms_handle_connect 0x7f591006db20 2024-03-29T11:35:24.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.583+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f593013e9b0 con 0x7f593013f650 2024-03-29T11:35:24.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.583+0000 7f591effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:24.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.583+0000 7f591effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:24.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.891+0000 7f5934b0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:24.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.891+0000 7f5934b0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:24.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.891+0000 7f5934b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:24.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.891+0000 7f5934b0e640 1 -- 172.21.15.37:0/357021486 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f593013e9b0 con 0x7f593013f650 2024-03-29T11:35:24.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.892+0000 7f591effd640 1 -- 172.21.15.37:0/357021486 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f593013e9b0 con 0x7f593013f650 2024-03-29T11:35:24.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.892+0000 7f591effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:24.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.892+0000 7f591effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:24.891 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:35:24.891 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:24.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.896+0000 7f5934b0e640 10 librados: watch_flush enter 2024-03-29T11:35:24.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.896+0000 7f5934b0e640 10 librados: watch_flush exit 2024-03-29T11:35:24.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.896+0000 7f5934b0e640 20 client.4244.objecter shutdown clearing up homeless session... 2024-03-29T11:35:24.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.896+0000 7f5934b0e640 10 client.4244.objecter successfully canceled tick 2024-03-29T11:35:24.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.896+0000 7f5934b0e640 1 -- 172.21.15.37:0/357021486 >> v1:172.21.15.146:6816/2899149139 conn(0x7f591006db20 legacy=0x7f591006ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:24.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.896+0000 7f5934b0e640 10 monclient: shutdown 2024-03-29T11:35:24.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.896+0000 7f5934b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:24.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.896+0000 7f5934b0e640 1 -- 172.21.15.37:0/357021486 >> v1:172.21.15.37:6790/0 conn(0x7f593013f650 legacy=0x7f593013c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:24.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.896+0000 7f5934b0e640 1 -- 172.21.15.37:0/357021486 shutdown_connections 2024-03-29T11:35:24.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.896+0000 7f5934b0e640 1 -- 172.21.15.37:0/357021486 >> 172.21.15.37:0/357021486 conn(0x7f5930066a00 msgr2=0x7f59300b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:24.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.897+0000 7f5934b0e640 1 -- 172.21.15.37:0/357021486 shutdown_connections 2024-03-29T11:35:24.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.897+0000 7f5934b0e640 1 -- 172.21.15.37:0/357021486 wait complete. 2024-03-29T11:35:24.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:24.897+0000 7f5934b0e640 1 librados: shutdown 2024-03-29T11:35:24.910 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:31.717 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:31.876 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.877+0000 7f186b863640 10 monclient: get_monmap_and_config 2024-03-29T11:35:31.876 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.877+0000 7f186b863640 10 monclient: build_initial_monmap 2024-03-29T11:35:31.876 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.877+0000 7f186b863640 10 monclient: monmap: 2024-03-29T11:35:31.876 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:31.876 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:31.876 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:35:31.878178+0000 2024-03-29T11:35:31.876 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:35:31.878178+0000 2024-03-29T11:35:31.876 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:31.876 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:31.876 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:31.877 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:31.877 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:31.877 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:31.877 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.878+0000 7f186b863640 1 Processor -- start 2024-03-29T11:35:31.877 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.878+0000 7f186b863640 1 -- start start 2024-03-29T11:35:31.877 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.878+0000 7f186b863640 10 monclient: init 2024-03-29T11:35:31.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.879+0000 7f186b863640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:31.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.879+0000 7f186b863640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:31.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.879+0000 7f186b863640 10 monclient(hunting): picked mon.noname-a con 0x7f18641457f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:31.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.879+0000 7f186b863640 10 monclient(hunting): picked mon.noname-c con 0x7f186413f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:31.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.879+0000 7f186b863640 10 monclient(hunting): picked mon.noname-b con 0x7f18640a9d70 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:31.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.879+0000 7f186b863640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18640aaef0 con 0x7f18641457f0 2024-03-29T11:35:31.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.879+0000 7f186b863640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1864145bd0 con 0x7f186413f650 2024-03-29T11:35:31.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.879+0000 7f186b863640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1864145eb0 con 0x7f18640a9d70 2024-03-29T11:35:31.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.879+0000 7f186b863640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:31.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.879+0000 7f186b863640 10 monclient(hunting): authenticate will time out at 911.328772s 2024-03-29T11:35:31.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.883+0000 7f1868dfe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f186413f650 0x7f18640b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:53261/0 (socket says 172.21.15.37:53261) 2024-03-29T11:35:31.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.883+0000 7f1868dfe640 1 -- 172.21.15.37:0/3968278963 learned_addr learned my addr 172.21.15.37:0/3968278963 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:31.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.887+0000 7f1857fff640 1 -- 172.21.15.37:0/3968278963 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274706838 0 0) 0x7f1864145bd0 con 0x7f186413f650 2024-03-29T11:35:31.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.887+0000 7f1857fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:31.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.887+0000 7f1857fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:31.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.887+0000 7f1857fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:31.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.887+0000 7f1857fff640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:35:31.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.887+0000 7f1857fff640 1 -- 172.21.15.37:0/3968278963 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1850003520 con 0x7f186413f650 2024-03-29T11:35:31.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 1 -- 172.21.15.37:0/3968278963 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3150144245 0 0) 0x7f1850003520 con 0x7f186413f650 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 1 -- 172.21.15.37:0/3968278963 >> v1:172.21.15.37:6790/0 conn(0x7f18640a9d70 legacy=0x7f186414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 1 -- 172.21.15.37:0/3968278963 >> v1:172.21.15.37:6789/0 conn(0x7f18641457f0 legacy=0x7f186413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 1 monclient: found mon.noname-c 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 1 -- 172.21.15.37:0/3968278963 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1864146340 con 0x7f186413f650 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 10 monclient: _finish_auth 0 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f186b863640 5 monclient: authenticate success, global_id 4249 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f186b863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 1 -- 172.21.15.37:0/3968278963 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1858001670 con 0x7f186413f650 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:31.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f1857fff640 10 monclient: dump: 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.888+0000 7f186b863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.889+0000 7f1857fff640 1 -- 172.21.15.37:0/3968278963 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1858001900 con 0x7f186413f650 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.889+0000 7f1857fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.889+0000 7f1857fff640 1 -- 172.21.15.37:0/3968278963 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1858001cd0 con 0x7f186413f650 2024-03-29T11:35:31.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.889+0000 7f1857fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:31.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.889+0000 7f1857fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:31.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.889+0000 7f1857fff640 10 monclient: dump: 2024-03-29T11:35:31.889 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:31.889 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:31.889 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:31.889 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:31.889 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:31.889 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:31.889 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:31.889 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:31.889 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:31.889 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:31.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.889+0000 7f186b863640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:31.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.889+0000 7f186b863640 4 set_mon_vals no callback set 2024-03-29T11:35:31.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.890+0000 7f186b863640 10 monclient: shutdown 2024-03-29T11:35:31.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.890+0000 7f186b863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:31.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.890+0000 7f186b863640 1 -- 172.21.15.37:0/3968278963 >> v1:172.21.15.146:6789/0 conn(0x7f186413f650 legacy=0x7f18640b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:31.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.891+0000 7f186b863640 1 -- 172.21.15.37:0/3968278963 shutdown_connections 2024-03-29T11:35:31.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.891+0000 7f186b863640 1 -- 172.21.15.37:0/3968278963 >> 172.21.15.37:0/3968278963 conn(0x7f1864066a00 msgr2=0x7f1864066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:31.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.891+0000 7f186b863640 1 -- 172.21.15.37:0/3968278963 shutdown_connections 2024-03-29T11:35:31.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.892+0000 7f186b863640 1 -- 172.21.15.37:0/3968278963 wait complete. 2024-03-29T11:35:31.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.892+0000 7f186b863640 10 monclient: build_initial_monmap 2024-03-29T11:35:31.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.892+0000 7f186b863640 10 monclient: monmap: 2024-03-29T11:35:31.893 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:31.893 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.893+0000 7f186b863640 1 librados: starting msgr at 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.893+0000 7f186b863640 1 librados: starting objecter 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.893+0000 7f186b863640 1 Processor -- start 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.893+0000 7f186b863640 1 -- start start 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.893+0000 7f186b863640 1 librados: setting wanted keys 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.893+0000 7f186b863640 1 librados: calling monclient init 2024-03-29T11:35:31.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.893+0000 7f186b863640 10 monclient: init 2024-03-29T11:35:31.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.894+0000 7f186b863640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:31.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.894+0000 7f186b863640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:31.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.894+0000 7f186b863640 10 monclient(hunting): picked mon.noname-c con 0x7f18640a9d70 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:31.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.894+0000 7f186b863640 10 monclient(hunting): picked mon.noname-b con 0x7f186413f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:31.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.894+0000 7f186b863640 10 monclient(hunting): picked mon.noname-a con 0x7f18641457f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:31.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.894+0000 7f186b863640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186413e940 con 0x7f18641457f0 2024-03-29T11:35:31.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.894+0000 7f186b863640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18641435c0 con 0x7f186413f650 2024-03-29T11:35:31.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.894+0000 7f186b863640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18641438a0 con 0x7f18640a9d70 2024-03-29T11:35:31.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.894+0000 7f186b863640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:31.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.894+0000 7f186b863640 10 monclient(hunting): authenticate will time out at 911.343301s 2024-03-29T11:35:31.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.898+0000 7f18695ff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f18640a9d70 0x7f1864130e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:44603/0 (socket says 172.21.15.37:44603) 2024-03-29T11:35:31.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.898+0000 7f18695ff640 1 -- 172.21.15.37:0/2752684473 learned_addr learned my addr 172.21.15.37:0/2752684473 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:31.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 10 client.?.objecter ms_handle_connect 0x7f18640a9d70 2024-03-29T11:35:31.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:31.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670765490 0 0) 0x7f18641438a0 con 0x7f18640a9d70 2024-03-29T11:35:31.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:31.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:31.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:31.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:35:31.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1848004eb0 con 0x7f18640a9d70 2024-03-29T11:35:31.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3145595127 0 0) 0x7f1848004eb0 con 0x7f18640a9d70 2024-03-29T11:35:31.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18641438a0 con 0x7f18640a9d70 2024-03-29T11:35:31.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f185c004360 con 0x7f18640a9d70 2024-03-29T11:35:31.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:31.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:31.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:31.899 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:31.899 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:31.899 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:31.899 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:31.899 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:31.899 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:31.899 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:31.900 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:31.900 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:31.900 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:31.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.899+0000 7f1855ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:31.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.900+0000 7f1855ffb640 10 client.?.objecter ms_handle_connect 0x7f18641457f0 2024-03-29T11:35:31.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.900+0000 7f1855ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:31.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.900+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853237075 0 0) 0x7f186413e940 con 0x7f18641457f0 2024-03-29T11:35:31.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.900+0000 7f1855ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:31.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.900+0000 7f1855ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:31.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.900+0000 7f1855ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:31.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.900+0000 7f1855ffb640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:35:31.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.900+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1848004eb0 con 0x7f18641457f0 2024-03-29T11:35:31.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.901+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3085545834 0 0) 0x7f1848004eb0 con 0x7f18641457f0 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.901+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f186413e940 con 0x7f18641457f0 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.901+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1860001770 con 0x7f18641457f0 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.901+0000 7f1855ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.901+0000 7f1855ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.901+0000 7f1855ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:31.905 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:31.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.901+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2198952194 0 0) 0x7f186413e940 con 0x7f18641457f0 2024-03-29T11:35:31.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.901+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 >> v1:172.21.15.37:6790/0 conn(0x7f18640a9d70 legacy=0x7f1864130e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:31.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.901+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 >> v1:172.21.15.146:6789/0 conn(0x7f186413f650 legacy=0x7f18641e77d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:31.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.902+0000 7f1855ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:31.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.902+0000 7f1855ffb640 1 monclient: found mon.a 2024-03-29T11:35:31.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.902+0000 7f1855ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:31.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.902+0000 7f1855ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:31.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.902+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1864131570 con 0x7f18641457f0 2024-03-29T11:35:31.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.902+0000 7f1855ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:31.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.902+0000 7f1855ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:31.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.902+0000 7f1855ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:31.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.902+0000 7f186b863640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:35:31.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.902+0000 7f186b863640 10 monclient: _renew_subs 2024-03-29T11:35:31.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.902+0000 7f186b863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:31.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.902+0000 7f186b863640 1 -- 172.21.15.37:0/2752684473 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186413cb10 con 0x7f18641457f0 2024-03-29T11:35:31.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.903+0000 7f186b863640 10 client.4248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:31.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.903+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1860001c40 con 0x7f18641457f0 2024-03-29T11:35:31.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.903+0000 7f186b863640 10 monclient: _renew_subs 2024-03-29T11:35:31.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.903+0000 7f186b863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:31.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.903+0000 7f186b863640 1 -- 172.21.15.37:0/2752684473 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1864136fc0 con 0x7f18641457f0 2024-03-29T11:35:31.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.903+0000 7f186b863640 1 librados: init done 2024-03-29T11:35:31.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.904+0000 7f1855ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:31.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.904+0000 7f18577fe640 4 set_mon_vals no callback set 2024-03-29T11:35:31.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.905+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1860003790 con 0x7f18641457f0 2024-03-29T11:35:31.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.905+0000 7f1855ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:31.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.905+0000 7f1855ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:31.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.905+0000 7f1855ffb640 10 monclient: dump: 2024-03-29T11:35:31.908 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:31.908 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:31.908 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:31.908 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:31.908 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:31.908 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:31.908 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:31.908 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:31.908 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:31.908 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:31.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.907+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f18600039b0 con 0x7f18641457f0 2024-03-29T11:35:31.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.907+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f1860083690 con 0x7f18641457f0 2024-03-29T11:35:31.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.907+0000 7f18695ff640 10 monclient: get_auth_request con 0x7f184806df10 auth_method 0 2024-03-29T11:35:31.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.907+0000 7f1855ffb640 10 client.4248.objecter ms_dispatch 0x7f1864000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:31.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.907+0000 7f1855ffb640 3 client.4248.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:31.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.907+0000 7f1855ffb640 3 client.4248.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:31.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.908+0000 7f1855ffb640 20 client.4248.objecter dump_active .. 0 homeless 2024-03-29T11:35:31.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.908+0000 7f1855ffb640 10 client.4248.objecter ms_handle_connect 0x7f184806df10 2024-03-29T11:35:31.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.908+0000 7f186b863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:31.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.908+0000 7f186b863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:31.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.908+0000 7f186b863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:31.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.908+0000 7f186b863640 1 -- 172.21.15.37:0/2752684473 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1844005080 con 0x7f18641457f0 2024-03-29T11:35:31.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.917+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f186004ef70 con 0x7f18641457f0 2024-03-29T11:35:31.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.917+0000 7f1855ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:31.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:31.917+0000 7f1855ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:32.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.237+0000 7f186b863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:32.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.237+0000 7f186b863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:32.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.237+0000 7f186b863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:32.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.237+0000 7f186b863640 1 -- 172.21.15.37:0/2752684473 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1844005330 con 0x7f18641457f0 2024-03-29T11:35:32.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.239+0000 7f1855ffb640 1 -- 172.21.15.37:0/2752684473 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f186004ef70 con 0x7f18641457f0 2024-03-29T11:35:32.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.239+0000 7f1855ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:32.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.239+0000 7f1855ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:32.238 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:35:32.238 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:32.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.243+0000 7f186b863640 10 librados: watch_flush enter 2024-03-29T11:35:32.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.244+0000 7f186b863640 10 librados: watch_flush exit 2024-03-29T11:35:32.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.244+0000 7f186b863640 20 client.4248.objecter shutdown clearing up homeless session... 2024-03-29T11:35:32.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.244+0000 7f186b863640 10 client.4248.objecter successfully canceled tick 2024-03-29T11:35:32.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.245+0000 7f186b863640 1 -- 172.21.15.37:0/2752684473 >> v1:172.21.15.146:6816/2899149139 conn(0x7f184806df10 legacy=0x7f18480703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.245+0000 7f186b863640 10 monclient: shutdown 2024-03-29T11:35:32.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.245+0000 7f186b863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:32.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.245+0000 7f186b863640 1 -- 172.21.15.37:0/2752684473 >> v1:172.21.15.37:6789/0 conn(0x7f18641457f0 legacy=0x7f18641e7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.245+0000 7f186b863640 1 -- 172.21.15.37:0/2752684473 shutdown_connections 2024-03-29T11:35:32.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.245+0000 7f186b863640 1 -- 172.21.15.37:0/2752684473 >> 172.21.15.37:0/2752684473 conn(0x7f1864066a00 msgr2=0x7f18640b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:32.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.245+0000 7f186b863640 1 -- 172.21.15.37:0/2752684473 shutdown_connections 2024-03-29T11:35:32.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.245+0000 7f186b863640 1 -- 172.21.15.37:0/2752684473 wait complete. 2024-03-29T11:35:32.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:32.246+0000 7f186b863640 1 librados: shutdown 2024-03-29T11:35:32.262 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:39.069 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:39.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.216+0000 7fe4f2be6640 10 monclient: get_monmap_and_config 2024-03-29T11:35:39.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.216+0000 7fe4f2be6640 10 monclient: build_initial_monmap 2024-03-29T11:35:39.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.216+0000 7fe4f2be6640 10 monclient: monmap: 2024-03-29T11:35:39.216 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:39.216 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:39.216 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:35:39.217615+0000 2024-03-29T11:35:39.216 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:35:39.217615+0000 2024-03-29T11:35:39.217 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:39.217 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:39.217 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:39.217 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:39.217 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:39.217 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:39.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.218+0000 7fe4f2be6640 1 Processor -- start 2024-03-29T11:35:39.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.218+0000 7fe4f2be6640 1 -- start start 2024-03-29T11:35:39.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.218+0000 7fe4f2be6640 10 monclient: init 2024-03-29T11:35:39.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.219+0000 7fe4f2be6640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:39.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.219+0000 7fe4f2be6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:39.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.219+0000 7fe4f2be6640 10 monclient(hunting): picked mon.noname-c con 0x7fe4ec0b4100 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:39.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.219+0000 7fe4f2be6640 10 monclient(hunting): picked mon.noname-b con 0x7fe4ec13ee90 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:39.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.219+0000 7fe4f2be6640 10 monclient(hunting): picked mon.noname-a con 0x7fe4ec0af900 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:39.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.219+0000 7fe4f2be6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec0afd10 con 0x7fe4ec0af900 2024-03-29T11:35:39.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.219+0000 7fe4f2be6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec0af6f0 con 0x7fe4ec0b4100 2024-03-29T11:35:39.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.219+0000 7fe4f2be6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec0aac80 con 0x7fe4ec13ee90 2024-03-29T11:35:39.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.219+0000 7fe4f2be6640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:39.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.219+0000 7fe4f2be6640 10 monclient(hunting): authenticate will time out at 918.668113s 2024-03-29T11:35:39.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.222+0000 7fe4ebfff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fe4ec13ee90 0x7fe4ec13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:43851/0 (socket says 172.21.15.37:43851) 2024-03-29T11:35:39.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.222+0000 7fe4ebfff640 1 -- 172.21.15.37:0/2895082610 learned_addr learned my addr 172.21.15.37:0/2895082610 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:39.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 1 -- 172.21.15.37:0/2895082610 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138452827 0 0) 0x7fe4ec0aac80 con 0x7fe4ec13ee90 2024-03-29T11:35:39.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:39.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:39.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:39.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:35:39.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 1 -- 172.21.15.37:0/2895082610 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d4003520 con 0x7fe4ec13ee90 2024-03-29T11:35:39.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 1 -- 172.21.15.37:0/2895082610 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311782162 0 0) 0x7fe4ec0af6f0 con 0x7fe4ec0b4100 2024-03-29T11:35:39.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:39.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:39.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:39.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:35:39.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 1 -- 172.21.15.37:0/2895082610 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4ec0aac80 con 0x7fe4ec0b4100 2024-03-29T11:35:39.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 1 -- 172.21.15.37:0/2895082610 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2971227745 0 0) 0x7fe4d4003520 con 0x7fe4ec13ee90 2024-03-29T11:35:39.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 1 -- 172.21.15.37:0/2895082610 >> v1:172.21.15.146:6789/0 conn(0x7fe4ec0b4100 legacy=0x7fe4ec0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.224+0000 7fe4eb7fe640 1 -- 172.21.15.37:0/2895082610 >> v1:172.21.15.37:6789/0 conn(0x7fe4ec0af900 legacy=0x7fe4ec14be10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:39.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:39.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:35:39.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:39.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:35:39.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 1 -- 172.21.15.37:0/2895082610 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec0aaf60 con 0x7fe4ec13ee90 2024-03-29T11:35:39.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 1 -- 172.21.15.37:0/2895082610 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fe4e4001a10 con 0x7fe4ec13ee90 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 10 monclient: dump: 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:39.225 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4f2be6640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4f2be6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 1 -- 172.21.15.37:0/2895082610 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4e4001bf0 con 0x7fe4ec13ee90 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4f2be6640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4f2be6640 4 set_mon_vals no callback set 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 1 -- 172.21.15.37:0/2895082610 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fe4e4002000 con 0x7fe4ec13ee90 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.226+0000 7fe4eb7fe640 10 monclient: dump: 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:39.226 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:39.227 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:39.227 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:39.227 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:39.227 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:39.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.227+0000 7fe4f2be6640 10 monclient: shutdown 2024-03-29T11:35:39.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.227+0000 7fe4f2be6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:39.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.227+0000 7fe4f2be6640 1 -- 172.21.15.37:0/2895082610 >> v1:172.21.15.37:6790/0 conn(0x7fe4ec13ee90 legacy=0x7fe4ec13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.233+0000 7fe4f2be6640 1 -- 172.21.15.37:0/2895082610 shutdown_connections 2024-03-29T11:35:39.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.233+0000 7fe4f2be6640 1 -- 172.21.15.37:0/2895082610 >> 172.21.15.37:0/2895082610 conn(0x7fe4ec0b35e0 msgr2=0x7fe4ec0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:39.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.235+0000 7fe4f2be6640 1 -- 172.21.15.37:0/2895082610 shutdown_connections 2024-03-29T11:35:39.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.235+0000 7fe4f2be6640 1 -- 172.21.15.37:0/2895082610 wait complete. 2024-03-29T11:35:39.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.236+0000 7fe4f2be6640 10 monclient: build_initial_monmap 2024-03-29T11:35:39.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.236+0000 7fe4f2be6640 10 monclient: monmap: 2024-03-29T11:35:39.235 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:39.235 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:39.235 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:35:39.235 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:35:39.235 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:39.235 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:39.235 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:39.235 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:39.235 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:39.235 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:39.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.237+0000 7fe4f2be6640 1 librados: starting msgr at 2024-03-29T11:35:39.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.237+0000 7fe4f2be6640 1 librados: starting objecter 2024-03-29T11:35:39.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.237+0000 7fe4f2be6640 1 Processor -- start 2024-03-29T11:35:39.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.237+0000 7fe4f2be6640 1 -- start start 2024-03-29T11:35:39.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.237+0000 7fe4f2be6640 1 librados: setting wanted keys 2024-03-29T11:35:39.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.237+0000 7fe4f2be6640 1 librados: calling monclient init 2024-03-29T11:35:39.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.237+0000 7fe4f2be6640 10 monclient: init 2024-03-29T11:35:39.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.238+0000 7fe4f2be6640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:39.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.238+0000 7fe4f2be6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:39.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.238+0000 7fe4f2be6640 10 monclient(hunting): picked mon.noname-b con 0x7fe4ec0af900 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:39.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.238+0000 7fe4f2be6640 10 monclient(hunting): picked mon.noname-c con 0x7fe4ec0b4100 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:39.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.238+0000 7fe4f2be6640 10 monclient(hunting): picked mon.noname-a con 0x7fe4ec13ee90 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:39.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.238+0000 7fe4f2be6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec0ac4a0 con 0x7fe4ec13ee90 2024-03-29T11:35:39.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.238+0000 7fe4f2be6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec0ac780 con 0x7fe4ec0af900 2024-03-29T11:35:39.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.238+0000 7fe4f2be6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec13b7b0 con 0x7fe4ec0b4100 2024-03-29T11:35:39.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.238+0000 7fe4f2be6640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:39.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.238+0000 7fe4f2be6640 10 monclient(hunting): authenticate will time out at 918.687310s 2024-03-29T11:35:39.240 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.241+0000 7fe4f0982640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe4ec0af900 0x7fe4ec0acf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:34171/0 (socket says 172.21.15.37:34171) 2024-03-29T11:35:39.240 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.241+0000 7fe4f0982640 1 -- 172.21.15.37:0/1619706270 learned_addr learned my addr 172.21.15.37:0/1619706270 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:39.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.244+0000 7fe4e97fa640 10 client.?.objecter ms_handle_connect 0x7fe4ec0af900 2024-03-29T11:35:39.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.244+0000 7fe4e97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:39.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.244+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53886150 0 0) 0x7fe4ec0ac780 con 0x7fe4ec0af900 2024-03-29T11:35:39.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.244+0000 7fe4e97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:39.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.244+0000 7fe4e97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.244+0000 7fe4e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.244+0000 7fe4e97fa640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.244+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d8004eb0 con 0x7fe4ec0af900 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.244+0000 7fe4e97fa640 10 client.?.objecter ms_handle_connect 0x7fe4ec0b4100 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.244+0000 7fe4e97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 132520547 0 0) 0x7fe4d8004eb0 con 0x7fe4ec0af900 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4ec0ac780 con 0x7fe4ec0af900 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fe4dc001a10 con 0x7fe4ec0af900 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 10 monclient(hunting): dump: 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:39.247 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:39.248 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:39.248 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:39.248 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:39.248 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:39.248 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:39.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:39.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265258408 0 0) 0x7fe4ec13b7b0 con 0x7fe4ec0b4100 2024-03-29T11:35:39.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:39.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:39.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:39.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:35:39.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d8004eb0 con 0x7fe4ec0b4100 2024-03-29T11:35:39.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3954631926 0 0) 0x7fe4ec0ac780 con 0x7fe4ec0af900 2024-03-29T11:35:39.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 >> v1:172.21.15.37:6790/0 conn(0x7fe4ec0b4100 legacy=0x7fe4ec1dc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 >> v1:172.21.15.37:6789/0 conn(0x7fe4ec13ee90 legacy=0x7fe4ec1e7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 1 monclient: found mon.b 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.245+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec13b410 con 0x7fe4ec0af900 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4e97fa640 10 monclient: _finish_auth 0 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4e97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4f2be6640 5 monclient: authenticate success, global_id 4255 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4f2be6640 10 monclient: _renew_subs 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4f2be6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4f2be6640 1 -- 172.21.15.37:0/1619706270 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ec0ad630 con 0x7fe4ec0af900 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4f2be6640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4f2be6640 10 monclient: _renew_subs 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4f2be6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4f2be6640 1 -- 172.21.15.37:0/1619706270 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ec1dbac0 con 0x7fe4ec0af900 2024-03-29T11:35:39.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4f2be6640 1 librados: init done 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4dc001c30 con 0x7fe4ec0af900 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4e97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fe4dc003b20 con 0x7fe4ec0af900 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4e97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.246+0000 7fe4e97fa640 10 monclient: dump: 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:39.250 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:39.251 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:39.251 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.247+0000 7fe4ea7fc640 4 set_mon_vals no callback set 2024-03-29T11:35:39.252 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.249+0000 7fe4f2be6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:39.252 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.249+0000 7fe4f2be6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:39.252 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.249+0000 7fe4f2be6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:39.252 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.249+0000 7fe4f2be6640 1 -- 172.21.15.37:0/1619706270 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4cc005080 con 0x7fe4ec0af900 2024-03-29T11:35:39.252 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.249+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fe4dc003c80 con 0x7fe4ec0af900 2024-03-29T11:35:39.252 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.250+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fe4dc084c30 con 0x7fe4ec0af900 2024-03-29T11:35:39.252 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.250+0000 7fe4e97fa640 10 client.4255.objecter ms_dispatch 0x7fe4ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:39.252 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.250+0000 7fe4e97fa640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:39.252 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.250+0000 7fe4e97fa640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:39.252 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.250+0000 7fe4e97fa640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-29T11:35:39.252 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.250+0000 7fe4ebfff640 10 monclient: get_auth_request con 0x7fe4d806ddf0 auth_method 0 2024-03-29T11:35:39.253 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.250+0000 7fe4e97fa640 10 client.4255.objecter ms_handle_connect 0x7fe4d806ddf0 2024-03-29T11:35:39.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.259+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4dc050510 con 0x7fe4ec0af900 2024-03-29T11:35:39.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.259+0000 7fe4e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:39.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.259+0000 7fe4e97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:39.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.582+0000 7fe4f2be6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:39.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.582+0000 7fe4f2be6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:39.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.582+0000 7fe4f2be6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:35:39.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.582+0000 7fe4f2be6640 1 -- 172.21.15.37:0/1619706270 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4cc005330 con 0x7fe4ec0af900 2024-03-29T11:35:39.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.583+0000 7fe4e97fa640 1 -- 172.21.15.37:0/1619706270 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fe4dc052440 con 0x7fe4ec0af900 2024-03-29T11:35:39.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.583+0000 7fe4e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:39.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.583+0000 7fe4e97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:39.582 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:35:39.582 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:39.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.588+0000 7fe4f2be6640 10 librados: watch_flush enter 2024-03-29T11:35:39.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.588+0000 7fe4f2be6640 10 librados: watch_flush exit 2024-03-29T11:35:39.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.588+0000 7fe4f2be6640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-29T11:35:39.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.588+0000 7fe4f2be6640 10 client.4255.objecter successfully canceled tick 2024-03-29T11:35:39.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.588+0000 7fe4f2be6640 1 -- 172.21.15.37:0/1619706270 >> v1:172.21.15.146:6816/2899149139 conn(0x7fe4d806ddf0 legacy=0x7fe4d80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.588+0000 7fe4f2be6640 10 monclient: shutdown 2024-03-29T11:35:39.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.588+0000 7fe4f2be6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:39.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.588+0000 7fe4f2be6640 1 -- 172.21.15.37:0/1619706270 >> v1:172.21.15.146:6789/0 conn(0x7fe4ec0af900 legacy=0x7fe4ec0acf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.589+0000 7fe4f2be6640 1 -- 172.21.15.37:0/1619706270 shutdown_connections 2024-03-29T11:35:39.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.589+0000 7fe4f2be6640 1 -- 172.21.15.37:0/1619706270 >> 172.21.15.37:0/1619706270 conn(0x7fe4ec0b35e0 msgr2=0x7fe4ec1428b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:39.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.591+0000 7fe4f2be6640 1 -- 172.21.15.37:0/1619706270 shutdown_connections 2024-03-29T11:35:39.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.591+0000 7fe4f2be6640 1 -- 172.21.15.37:0/1619706270 wait complete. 2024-03-29T11:35:39.590 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:39.592+0000 7fe4f2be6640 1 librados: shutdown 2024-03-29T11:35:39.615 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:46.422 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:46.573 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.574+0000 7f37b63eb640 10 monclient: get_monmap_and_config 2024-03-29T11:35:46.573 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.574+0000 7f37b63eb640 10 monclient: build_initial_monmap 2024-03-29T11:35:46.573 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.574+0000 7f37b63eb640 10 monclient: monmap: 2024-03-29T11:35:46.573 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:46.573 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:46.573 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:35:46.575772+0000 2024-03-29T11:35:46.573 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:35:46.575772+0000 2024-03-29T11:35:46.573 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:46.574 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:46.574 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:46.574 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:46.574 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:46.574 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:46.574 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.575+0000 7f37b63eb640 1 Processor -- start 2024-03-29T11:35:46.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.576+0000 7f37b63eb640 1 -- start start 2024-03-29T11:35:46.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.576+0000 7f37b63eb640 10 monclient: init 2024-03-29T11:35:46.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.576+0000 7f37b63eb640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:46.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.576+0000 7f37b63eb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:46.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.576+0000 7f37b63eb640 10 monclient(hunting): picked mon.noname-b con 0x7f37b014a400 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:46.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.576+0000 7f37b63eb640 10 monclient(hunting): picked mon.noname-c con 0x7f37b00aa500 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:46.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.577+0000 7f37b63eb640 10 monclient(hunting): picked mon.noname-a con 0x7f37b0149800 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:46.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.577+0000 7f37b63eb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b00aca50 con 0x7f37b0149800 2024-03-29T11:35:46.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.577+0000 7f37b63eb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b0149cb0 con 0x7f37b00aa500 2024-03-29T11:35:46.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.577+0000 7f37b63eb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b00acc80 con 0x7f37b014a400 2024-03-29T11:35:46.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.577+0000 7f37b63eb640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:46.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.577+0000 7f37b63eb640 10 monclient(hunting): authenticate will time out at 926.025982s 2024-03-29T11:35:46.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.580+0000 7f37b4be8640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f37b00aa500 0x7f37b00ad160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:38797/0 (socket says 172.21.15.37:38797) 2024-03-29T11:35:46.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.580+0000 7f37b4be8640 1 -- 172.21.15.37:0/1082384682 learned_addr learned my addr 172.21.15.37:0/1082384682 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:46.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 1 -- 172.21.15.37:0/1082384682 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311104893 0 0) 0x7f37b0149cb0 con 0x7f37b00aa500 2024-03-29T11:35:46.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:46.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:46.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:46.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:35:46.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 1 -- 172.21.15.37:0/1082384682 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f379c003520 con 0x7f37b00aa500 2024-03-29T11:35:46.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 1 -- 172.21.15.37:0/1082384682 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480645109 0 0) 0x7f37b00aca50 con 0x7f37b0149800 2024-03-29T11:35:46.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:46.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 1 -- 172.21.15.37:0/1082384682 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b0149cb0 con 0x7f37b0149800 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 1 -- 172.21.15.37:0/1082384682 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2262117701 0 0) 0x7f379c003520 con 0x7f37b00aa500 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 1 -- 172.21.15.37:0/1082384682 >> v1:172.21.15.37:6790/0 conn(0x7f37b014a400 legacy=0x7f37b00aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 1 -- 172.21.15.37:0/1082384682 >> v1:172.21.15.37:6789/0 conn(0x7f37b0149800 legacy=0x7f37b0151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 1 monclient: found mon.noname-c 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 1 -- 172.21.15.37:0/1082384682 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b00ac350 con 0x7f37b00aa500 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient: _finish_auth 0 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 1 -- 172.21.15.37:0/1082384682 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f37a8001a10 con 0x7f37b00aa500 2024-03-29T11:35:46.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:46.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:46.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37a7fff640 10 monclient: dump: 2024-03-29T11:35:46.584 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:46.584 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:46.584 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:46.584 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:46.584 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:46.584 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:46.584 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:46.584 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:46.584 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:46.584 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:46.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37b63eb640 5 monclient: authenticate success, global_id 4258 2024-03-29T11:35:46.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.582+0000 7f37b63eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:46.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.583+0000 7f37a7fff640 1 -- 172.21.15.37:0/1082384682 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37a8001b70 con 0x7f37b00aa500 2024-03-29T11:35:46.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.583+0000 7f37a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:46.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.583+0000 7f37a7fff640 1 -- 172.21.15.37:0/1082384682 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f37a8001f80 con 0x7f37b00aa500 2024-03-29T11:35:46.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.583+0000 7f37a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:46.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.583+0000 7f37a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:46.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.583+0000 7f37a7fff640 10 monclient: dump: 2024-03-29T11:35:46.587 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:46.587 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:46.587 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:46.587 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:46.587 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:46.587 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:46.587 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:46.587 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:46.588 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:46.588 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:46.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.586+0000 7f37b63eb640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:46.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.586+0000 7f37b63eb640 4 set_mon_vals no callback set 2024-03-29T11:35:46.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.587+0000 7f37b63eb640 10 monclient: shutdown 2024-03-29T11:35:46.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.587+0000 7f37b63eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:46.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.587+0000 7f37b63eb640 1 -- 172.21.15.37:0/1082384682 >> v1:172.21.15.146:6789/0 conn(0x7f37b00aa500 legacy=0x7f37b00ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:46.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.587+0000 7f37b63eb640 1 -- 172.21.15.37:0/1082384682 shutdown_connections 2024-03-29T11:35:46.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.587+0000 7f37b63eb640 1 -- 172.21.15.37:0/1082384682 >> 172.21.15.37:0/1082384682 conn(0x7f37b00b3a30 msgr2=0x7f37b00b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:46.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.587+0000 7f37b63eb640 1 -- 172.21.15.37:0/1082384682 shutdown_connections 2024-03-29T11:35:46.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.587+0000 7f37b63eb640 1 -- 172.21.15.37:0/1082384682 wait complete. 2024-03-29T11:35:46.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.588+0000 7f37b63eb640 10 monclient: build_initial_monmap 2024-03-29T11:35:46.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.588+0000 7f37b63eb640 10 monclient: monmap: 2024-03-29T11:35:46.592 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:46.592 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:46.592 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:35:46.592 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:35:46.592 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:46.592 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:46.592 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:46.592 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:46.593 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:46.593 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:46.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 1 librados: starting msgr at 2024-03-29T11:35:46.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 1 librados: starting objecter 2024-03-29T11:35:46.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 1 Processor -- start 2024-03-29T11:35:46.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 1 -- start start 2024-03-29T11:35:46.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 1 librados: setting wanted keys 2024-03-29T11:35:46.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 1 librados: calling monclient init 2024-03-29T11:35:46.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 10 monclient: init 2024-03-29T11:35:46.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:46.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:46.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 10 monclient(hunting): picked mon.noname-c con 0x7f37b00aa500 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:46.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 10 monclient(hunting): picked mon.noname-b con 0x7f37b0149800 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:46.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 10 monclient(hunting): picked mon.noname-a con 0x7f37b014a400 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:46.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b0142360 con 0x7f37b014a400 2024-03-29T11:35:46.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b006efe0 con 0x7f37b0149800 2024-03-29T11:35:46.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b006f2c0 con 0x7f37b00aa500 2024-03-29T11:35:46.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:46.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.589+0000 7f37b63eb640 10 monclient(hunting): authenticate will time out at 926.038925s 2024-03-29T11:35:46.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.595+0000 7f37b4be8640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f37b0149800 0x7f37b0071ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:38869/0 (socket says 172.21.15.37:38869) 2024-03-29T11:35:46.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.595+0000 7f37b4be8640 1 -- 172.21.15.37:0/2393934492 learned_addr learned my addr 172.21.15.37:0/2393934492 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:46.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.596+0000 7f37a5ffb640 10 client.?.objecter ms_handle_connect 0x7f37b014a400 2024-03-29T11:35:46.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.596+0000 7f37a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:46.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.596+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864381342 0 0) 0x7f37b0142360 con 0x7f37b014a400 2024-03-29T11:35:46.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.596+0000 7f37a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:46.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.596+0000 7f37a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:46.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.596+0000 7f37a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:46.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.596+0000 7f37a5ffb640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:35:46.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.596+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3794004eb0 con 0x7f37b014a400 2024-03-29T11:35:46.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.596+0000 7f37a5ffb640 10 client.?.objecter ms_handle_connect 0x7f37b0149800 2024-03-29T11:35:46.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.596+0000 7f37a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:46.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854280672 0 0) 0x7f37b006efe0 con 0x7f37b0149800 2024-03-29T11:35:46.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:46.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:46.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:46.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:35:46.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37940051d0 con 0x7f37b0149800 2024-03-29T11:35:46.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2330698999 0 0) 0x7f3794004eb0 con 0x7f37b014a400 2024-03-29T11:35:46.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3794005640 con 0x7f37b014a400 2024-03-29T11:35:46.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f37ac0018b0 con 0x7f37b014a400 2024-03-29T11:35:46.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:46.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:46.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:46.602 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:46.602 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:46.602 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:46.602 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:46.602 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3787469664 0 0) 0x7f37940051d0 con 0x7f37b0149800 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.597+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3794005d30 con 0x7f37b0149800 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f37a8001ec0 con 0x7f37b0149800 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:46.603 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 416450771 0 0) 0x7f3794005640 con 0x7f37b014a400 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 >> v1:172.21.15.37:6790/0 conn(0x7f37b00aa500 legacy=0x7f37b0140d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 >> v1:172.21.15.146:6789/0 conn(0x7f37b0149800 legacy=0x7f37b0071ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 1 monclient: found mon.a 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b006f5a0 con 0x7f37b014a400 2024-03-29T11:35:46.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:46.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:46.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.598+0000 7f37a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:46.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.600+0000 7f37b63eb640 5 monclient: authenticate success, global_id 4254 2024-03-29T11:35:46.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.600+0000 7f37b63eb640 10 monclient: _renew_subs 2024-03-29T11:35:46.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.600+0000 7f37b63eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:46.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.600+0000 7f37b63eb640 1 -- 172.21.15.37:0/2393934492 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b006dd50 con 0x7f37b014a400 2024-03-29T11:35:46.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.600+0000 7f37b63eb640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:46.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.600+0000 7f37b63eb640 10 monclient: _renew_subs 2024-03-29T11:35:46.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.600+0000 7f37b63eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:46.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.600+0000 7f37b63eb640 1 -- 172.21.15.37:0/2393934492 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b006d7c0 con 0x7f37b014a400 2024-03-29T11:35:46.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.600+0000 7f37b63eb640 1 librados: init done 2024-03-29T11:35:46.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.602+0000 7f379b7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:46.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.602+0000 7f379b7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:46.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.602+0000 7f379b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:46.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.602+0000 7f379b7fe640 1 -- 172.21.15.37:0/2393934492 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b006df90 con 0x7f37b014a400 2024-03-29T11:35:46.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.603+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37ac0035f0 con 0x7f37b014a400 2024-03-29T11:35:46.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.603+0000 7f37a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:46.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.603+0000 7f37a77fe640 4 set_mon_vals no callback set 2024-03-29T11:35:46.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.604+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f37ac0018b0 con 0x7f37b014a400 2024-03-29T11:35:46.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.604+0000 7f37a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:46.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.604+0000 7f37a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:46.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.604+0000 7f37a5ffb640 10 monclient: dump: 2024-03-29T11:35:46.606 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:46.607 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:46.607 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:46.607 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:46.607 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:46.607 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:46.607 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:46.607 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:46.607 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:46.607 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:46.608 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.610+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f37ac0195d0 con 0x7f37b014a400 2024-03-29T11:35:46.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.611+0000 7f37b53e9640 10 monclient: get_auth_request con 0x7f379406e720 auth_method 0 2024-03-29T11:35:46.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.611+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f37ac083370 con 0x7f37b014a400 2024-03-29T11:35:46.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.611+0000 7f37a5ffb640 10 client.4254.objecter ms_dispatch 0x7f37b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:46.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.611+0000 7f37a5ffb640 3 client.4254.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:46.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.611+0000 7f37a5ffb640 3 client.4254.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:46.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.611+0000 7f37a5ffb640 20 client.4254.objecter dump_active .. 0 homeless 2024-03-29T11:35:46.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.611+0000 7f37a5ffb640 10 client.4254.objecter ms_handle_connect 0x7f379406e720 2024-03-29T11:35:46.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.612+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37ac04ec50 con 0x7f37b014a400 2024-03-29T11:35:46.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.612+0000 7f37a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:46.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.612+0000 7f37a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:46.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.941+0000 7f379b7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:46.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.941+0000 7f379b7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:46.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.941+0000 7f379b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:35:46.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.941+0000 7f379b7fe640 1 -- 172.21.15.37:0/2393934492 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37b004f730 con 0x7f37b014a400 2024-03-29T11:35:46.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.942+0000 7f37a5ffb640 1 -- 172.21.15.37:0/2393934492 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f37ac04ec50 con 0x7f37b014a400 2024-03-29T11:35:46.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.942+0000 7f37a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:46.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.942+0000 7f37a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:46.946 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:35:46.946 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:46.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.950+0000 7f379b7fe640 10 librados: watch_flush enter 2024-03-29T11:35:46.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.950+0000 7f379b7fe640 10 librados: watch_flush exit 2024-03-29T11:35:46.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.950+0000 7f379b7fe640 20 client.4254.objecter shutdown clearing up homeless session... 2024-03-29T11:35:46.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.950+0000 7f379b7fe640 10 client.4254.objecter successfully canceled tick 2024-03-29T11:35:46.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.950+0000 7f379b7fe640 1 -- 172.21.15.37:0/2393934492 >> v1:172.21.15.146:6816/2899149139 conn(0x7f379406e720 legacy=0x7f3794070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:46.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.950+0000 7f379b7fe640 10 monclient: shutdown 2024-03-29T11:35:46.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.950+0000 7f379b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:46.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.950+0000 7f379b7fe640 1 -- 172.21.15.37:0/2393934492 >> v1:172.21.15.37:6789/0 conn(0x7f37b014a400 legacy=0x7f37b00721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:46.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.951+0000 7f379b7fe640 1 -- 172.21.15.37:0/2393934492 shutdown_connections 2024-03-29T11:35:46.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.951+0000 7f379b7fe640 1 -- 172.21.15.37:0/2393934492 >> 172.21.15.37:0/2393934492 conn(0x7f37b00b3a30 msgr2=0x7f37b00b1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:46.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.951+0000 7f379b7fe640 1 -- 172.21.15.37:0/2393934492 shutdown_connections 2024-03-29T11:35:46.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.951+0000 7f379b7fe640 1 -- 172.21.15.37:0/2393934492 wait complete. 2024-03-29T11:35:46.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:46.952+0000 7f379b7fe640 1 librados: shutdown 2024-03-29T11:35:46.967 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:53.774 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:53.925 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.927+0000 7f04e2658640 10 monclient: get_monmap_and_config 2024-03-29T11:35:53.925 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.927+0000 7f04e2658640 10 monclient: build_initial_monmap 2024-03-29T11:35:53.925 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.927+0000 7f04e2658640 10 monclient: monmap: 2024-03-29T11:35:53.926 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:53.926 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:53.926 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:35:53.928103+0000 2024-03-29T11:35:53.926 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:35:53.928103+0000 2024-03-29T11:35:53.926 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:53.926 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:53.926 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:53.926 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:35:53.926 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:35:53.926 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:53.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.928+0000 7f04e2658640 1 Processor -- start 2024-03-29T11:35:53.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.928+0000 7f04e2658640 1 -- start start 2024-03-29T11:35:53.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.928+0000 7f04e2658640 10 monclient: init 2024-03-29T11:35:53.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.929+0000 7f04e2658640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:53.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.929+0000 7f04e2658640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:53.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.930+0000 7f04e2658640 10 monclient(hunting): picked mon.noname-a con 0x7f04dc0b2800 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:53.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.930+0000 7f04e2658640 10 monclient(hunting): picked mon.noname-b con 0x7f04dc0b3220 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:53.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.930+0000 7f04e2658640 10 monclient(hunting): picked mon.noname-c con 0x7f04dc1439c0 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:53.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.930+0000 7f04e2658640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc0b3840 con 0x7f04dc0b2800 2024-03-29T11:35:53.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.930+0000 7f04e2658640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc06f230 con 0x7f04dc1439c0 2024-03-29T11:35:53.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.930+0000 7f04e2658640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc06ef50 con 0x7f04dc0b3220 2024-03-29T11:35:53.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.930+0000 7f04e2658640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:53.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.930+0000 7f04e2658640 10 monclient(hunting): authenticate will time out at 933.379562s 2024-03-29T11:35:53.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.934+0000 7f04e0e55640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f04dc0b3220 0x7f04dc146710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:43255/0 (socket says 172.21.15.37:43255) 2024-03-29T11:35:53.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.934+0000 7f04e0e55640 1 -- 172.21.15.37:0/1074144029 learned_addr learned my addr 172.21.15.37:0/1074144029 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:53.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.935+0000 7f04d3fff640 1 -- 172.21.15.37:0/1074144029 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119977712 0 0) 0x7f04dc06ef50 con 0x7f04dc0b3220 2024-03-29T11:35:53.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.935+0000 7f04d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:53.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.935+0000 7f04d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:53.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.935+0000 7f04d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:53.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.935+0000 7f04d3fff640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:35:53.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.935+0000 7f04d3fff640 1 -- 172.21.15.37:0/1074144029 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04c8003520 con 0x7f04dc0b3220 2024-03-29T11:35:53.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.935+0000 7f04d3fff640 1 -- 172.21.15.37:0/1074144029 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3277675619 0 0) 0x7f04c8003520 con 0x7f04dc0b3220 2024-03-29T11:35:53.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.935+0000 7f04d3fff640 1 -- 172.21.15.37:0/1074144029 >> v1:172.21.15.146:6789/0 conn(0x7f04dc1439c0 legacy=0x7f04dc151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:53.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.935+0000 7f04d3fff640 1 -- 172.21.15.37:0/1074144029 >> v1:172.21.15.37:6789/0 conn(0x7f04dc0b2800 legacy=0x7f04dc1432b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:53.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.936+0000 7f04d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:53.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.936+0000 7f04d3fff640 1 monclient: found mon.noname-b 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.936+0000 7f04d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.936+0000 7f04d3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.936+0000 7f04d3fff640 1 -- 172.21.15.37:0/1074144029 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc071d30 con 0x7f04dc0b3220 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.936+0000 7f04d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.936+0000 7f04d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.936+0000 7f04d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.936+0000 7f04d3fff640 1 -- 172.21.15.37:0/1074144029 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f04d8001670 con 0x7f04dc0b3220 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.936+0000 7f04d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.936+0000 7f04d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.936+0000 7f04d3fff640 10 monclient: dump: 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:53.936 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:53.937 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:53.937 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:53.937 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:53.937 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:53.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.938+0000 7f04e2658640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:35:53.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.938+0000 7f04e2658640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:53.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.938+0000 7f04d3fff640 1 -- 172.21.15.37:0/1074144029 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04d80018d0 con 0x7f04dc0b3220 2024-03-29T11:35:53.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.938+0000 7f04d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:53.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.938+0000 7f04e2658640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:53.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.938+0000 7f04e2658640 4 set_mon_vals no callback set 2024-03-29T11:35:53.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.939+0000 7f04d3fff640 1 -- 172.21.15.37:0/1074144029 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f04d8001d20 con 0x7f04dc0b3220 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.939+0000 7f04d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.939+0000 7f04d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.939+0000 7f04d3fff640 10 monclient: dump: 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.939+0000 7f04e2658640 10 monclient: shutdown 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.939+0000 7f04e2658640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:53.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.939+0000 7f04e2658640 1 -- 172.21.15.37:0/1074144029 >> v1:172.21.15.37:6790/0 conn(0x7f04dc0b3220 legacy=0x7f04dc146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:53.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.939+0000 7f04e2658640 1 -- 172.21.15.37:0/1074144029 shutdown_connections 2024-03-29T11:35:53.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.939+0000 7f04e2658640 1 -- 172.21.15.37:0/1074144029 >> 172.21.15.37:0/1074144029 conn(0x7f04dc066a00 msgr2=0x7f04dc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:53.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.940+0000 7f04e2658640 1 -- 172.21.15.37:0/1074144029 shutdown_connections 2024-03-29T11:35:53.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.940+0000 7f04e2658640 1 -- 172.21.15.37:0/1074144029 wait complete. 2024-03-29T11:35:53.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.940+0000 7f04e2658640 10 monclient: build_initial_monmap 2024-03-29T11:35:53.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.940+0000 7f04e2658640 10 monclient: monmap: 2024-03-29T11:35:53.941 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:35:53.941 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:53.941 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:35:53.941 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:35:53.941 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:53.941 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:53.941 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:35:53.941 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:35:53.941 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:35:53.941 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:53.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.941+0000 7f04e2658640 1 librados: starting msgr at 2024-03-29T11:35:53.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.941+0000 7f04e2658640 1 librados: starting objecter 2024-03-29T11:35:53.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.941+0000 7f04e2658640 1 Processor -- start 2024-03-29T11:35:53.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.941+0000 7f04e2658640 1 -- start start 2024-03-29T11:35:53.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.941+0000 7f04e2658640 1 librados: setting wanted keys 2024-03-29T11:35:53.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.941+0000 7f04e2658640 1 librados: calling monclient init 2024-03-29T11:35:53.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.941+0000 7f04e2658640 10 monclient: init 2024-03-29T11:35:53.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.942+0000 7f04e2658640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:53.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.942+0000 7f04e2658640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:53.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.942+0000 7f04e2658640 10 monclient(hunting): picked mon.noname-a con 0x7f04dc0b2800 addr v1:172.21.15.37:6789/0 2024-03-29T11:35:53.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.942+0000 7f04e2658640 10 monclient(hunting): picked mon.noname-b con 0x7f04dc1439c0 addr v1:172.21.15.146:6789/0 2024-03-29T11:35:53.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.942+0000 7f04e2658640 10 monclient(hunting): picked mon.noname-c con 0x7f04dc13e0d0 addr v1:172.21.15.37:6790/0 2024-03-29T11:35:53.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.942+0000 7f04e2658640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc13df20 con 0x7f04dc0b2800 2024-03-29T11:35:53.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.942+0000 7f04e2658640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc13e680 con 0x7f04dc1439c0 2024-03-29T11:35:53.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.942+0000 7f04e2658640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc13eb10 con 0x7f04dc13e0d0 2024-03-29T11:35:53.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.942+0000 7f04e2658640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:53.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.942+0000 7f04e2658640 10 monclient(hunting): authenticate will time out at 933.391499s 2024-03-29T11:35:53.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.945+0000 7f04e1e57640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f04dc13e0d0 0x7f04dc1e62b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:43347/0 (socket says 172.21.15.37:43347) 2024-03-29T11:35:53.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.945+0000 7f04e1e57640 1 -- 172.21.15.37:0/3447152504 learned_addr learned my addr 172.21.15.37:0/3447152504 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:35:53.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.946+0000 7f04d1ffb640 10 client.?.objecter ms_handle_connect 0x7f04dc13e0d0 2024-03-29T11:35:53.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.947+0000 7f04d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:53.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.947+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992028646 0 0) 0x7f04dc13eb10 con 0x7f04dc13e0d0 2024-03-29T11:35:53.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.947+0000 7f04d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:53.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.947+0000 7f04d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:53.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.947+0000 7f04d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:53.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.947+0000 7f04d1ffb640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:35:53.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.947+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04c0004eb0 con 0x7f04dc13e0d0 2024-03-29T11:35:53.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.947+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2941039167 0 0) 0x7f04c0004eb0 con 0x7f04dc13e0d0 2024-03-29T11:35:53.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.947+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04dc13eb10 con 0x7f04dc13e0d0 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.947+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f04d4001ad0 con 0x7f04dc13e0d0 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.947+0000 7f04d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.947+0000 7f04d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.948+0000 7f04d1ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:53.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.948+0000 7f04d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:53.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.948+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1114234541 0 0) 0x7f04dc13eb10 con 0x7f04dc13e0d0 2024-03-29T11:35:53.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.948+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 >> v1:172.21.15.146:6789/0 conn(0x7f04dc1439c0 legacy=0x7f04dc1db800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:53.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.950+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 >> v1:172.21.15.37:6789/0 conn(0x7f04dc0b2800 legacy=0x7f04dc13d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:53.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:53.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 1 monclient: found mon.c 2024-03-29T11:35:53.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:53.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:53.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc13f920 con 0x7f04dc13e0d0 2024-03-29T11:35:53.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:53.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:53.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:53.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04e2658640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04e2658640 10 monclient: _renew_subs 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04e2658640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04e2658640 1 -- 172.21.15.37:0/3447152504 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04dc1dbf10 con 0x7f04dc13e0d0 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04e2658640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04e2658640 10 monclient: _renew_subs 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04e2658640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04e2658640 1 -- 172.21.15.37:0/3447152504 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04dc1dc2b0 con 0x7f04dc13e0d0 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04e2658640 1 librados: init done 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 10 client.4265.objecter ms_handle_connect 0x7f04dc0b2800 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 10 client.4265.objecter resend_mon_ops 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04d4001d30 con 0x7f04dc13e0d0 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f04d4004dc0 con 0x7f04dc13e0d0 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:53.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d1ffb640 10 monclient: dump: 2024-03-29T11:35:53.952 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:35:53.952 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:35:53.952 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:53.952 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:35:53.952 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:53.952 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:35:53.952 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:35:53.952 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:35:53.952 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:35:53.952 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:35:53.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.951+0000 7f04d37fe640 4 set_mon_vals no callback set 2024-03-29T11:35:53.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.953+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f04d401ad90 con 0x7f04dc13e0d0 2024-03-29T11:35:53.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.953+0000 7f04e2658640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:53.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.953+0000 7f04e2658640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:53.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.953+0000 7f04e2658640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:53.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.953+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f04d4084de0 con 0x7f04dc13e0d0 2024-03-29T11:35:53.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.953+0000 7f04d1ffb640 10 client.4265.objecter ms_dispatch 0x7f04dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:53.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.953+0000 7f04d1ffb640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:53.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.953+0000 7f04d1ffb640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:53.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.953+0000 7f04e2658640 1 -- 172.21.15.37:0/3447152504 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04bc005080 con 0x7f04dc13e0d0 2024-03-29T11:35:53.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.954+0000 7f04d1ffb640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-29T11:35:53.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.957+0000 7f04e1656640 10 monclient: get_auth_request con 0x7f04c006db60 auth_method 0 2024-03-29T11:35:53.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.960+0000 7f04d1ffb640 10 client.4265.objecter ms_handle_connect 0x7f04c006db60 2024-03-29T11:35:53.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.961+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04d40504f0 con 0x7f04dc13e0d0 2024-03-29T11:35:53.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.961+0000 7f04d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:53.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:53.961+0000 7f04d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:54.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.286+0000 7f04e2658640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:54.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.287+0000 7f04e2658640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:54.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.287+0000 7f04e2658640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:35:54.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.287+0000 7f04e2658640 1 -- 172.21.15.37:0/3447152504 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04bc005330 con 0x7f04dc13e0d0 2024-03-29T11:35:54.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.287+0000 7f04d1ffb640 1 -- 172.21.15.37:0/3447152504 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f04d40525f0 con 0x7f04dc13e0d0 2024-03-29T11:35:54.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.287+0000 7f04d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:54.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.287+0000 7f04d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:54.288 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:35:54.288 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:54.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.295+0000 7f04e2658640 10 librados: watch_flush enter 2024-03-29T11:35:54.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.295+0000 7f04e2658640 10 librados: watch_flush exit 2024-03-29T11:35:54.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.295+0000 7f04e2658640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-29T11:35:54.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.295+0000 7f04e2658640 10 client.4265.objecter successfully canceled tick 2024-03-29T11:35:54.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.295+0000 7f04e2658640 1 -- 172.21.15.37:0/3447152504 >> v1:172.21.15.146:6816/2899149139 conn(0x7f04c006db60 legacy=0x7f04c0070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.295+0000 7f04e2658640 10 monclient: shutdown 2024-03-29T11:35:54.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.295+0000 7f04e2658640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:54.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.295+0000 7f04e2658640 1 -- 172.21.15.37:0/3447152504 >> v1:172.21.15.37:6790/0 conn(0x7f04dc13e0d0 legacy=0x7f04dc1e62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.296+0000 7f04e2658640 1 -- 172.21.15.37:0/3447152504 shutdown_connections 2024-03-29T11:35:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.296+0000 7f04e2658640 1 -- 172.21.15.37:0/3447152504 >> 172.21.15.37:0/3447152504 conn(0x7f04dc066a00 msgr2=0x7f04dc140360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.296+0000 7f04e2658640 1 -- 172.21.15.37:0/3447152504 shutdown_connections 2024-03-29T11:35:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.296+0000 7f04e2658640 1 -- 172.21.15.37:0/3447152504 wait complete. 2024-03-29T11:35:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:35:54.297+0000 7f04e2658640 1 librados: shutdown 2024-03-29T11:35:54.312 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:01.118 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:01.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.268+0000 7f3486974640 10 monclient: get_monmap_and_config 2024-03-29T11:36:01.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.268+0000 7f3486974640 10 monclient: build_initial_monmap 2024-03-29T11:36:01.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.268+0000 7f3486974640 10 monclient: monmap: 2024-03-29T11:36:01.266 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:01.266 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:01.267 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:36:01.268957+0000 2024-03-29T11:36:01.267 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:36:01.268957+0000 2024-03-29T11:36:01.267 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:01.267 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:01.267 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:01.267 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:36:01.267 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:36:01.267 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:01.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.269+0000 7f3486974640 1 Processor -- start 2024-03-29T11:36:01.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.269+0000 7f3486974640 1 -- start start 2024-03-29T11:36:01.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.269+0000 7f3486974640 10 monclient: init 2024-03-29T11:36:01.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.270+0000 7f3486974640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:01.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.270+0000 7f3486974640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:01.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.270+0000 7f3486974640 10 monclient(hunting): picked mon.noname-b con 0x7f34800b4820 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:01.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.270+0000 7f3486974640 10 monclient(hunting): picked mon.noname-c con 0x7f34800a9ce0 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:01.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.270+0000 7f3486974640 10 monclient(hunting): picked mon.noname-a con 0x7f34800b3cd0 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:01.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.270+0000 7f3486974640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34800b4180 con 0x7f34800b3cd0 2024-03-29T11:36:01.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.270+0000 7f3486974640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34800aaf60 con 0x7f34800a9ce0 2024-03-29T11:36:01.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.270+0000 7f3486974640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348013c200 con 0x7f34800b4820 2024-03-29T11:36:01.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.270+0000 7f3486974640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:01.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.270+0000 7f3486974640 10 monclient(hunting): authenticate will time out at 940.719866s 2024-03-29T11:36:01.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.275+0000 7f3485972640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f34800b4820 0x7f348013baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:47833/0 (socket says 172.21.15.37:47833) 2024-03-29T11:36:01.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.275+0000 7f3485972640 1 -- 172.21.15.37:0/2703769091 learned_addr learned my addr 172.21.15.37:0/2703769091 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:01.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.276+0000 7f3484970640 1 -- 172.21.15.37:0/2703769091 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178200762 0 0) 0x7f348013c200 con 0x7f34800b4820 2024-03-29T11:36:01.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.276+0000 7f3484970640 10 monclient(hunting): _negotiate 2024-03-29T11:36:01.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.276+0000 7f3484970640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:01.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.276+0000 7f3484970640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:01.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.276+0000 7f3484970640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:36:01.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.276+0000 7f3484970640 1 -- 172.21.15.37:0/2703769091 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f346c003520 con 0x7f34800b4820 2024-03-29T11:36:01.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.277+0000 7f3484970640 1 -- 172.21.15.37:0/2703769091 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3555002370 0 0) 0x7f346c003520 con 0x7f34800b4820 2024-03-29T11:36:01.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.277+0000 7f3484970640 1 -- 172.21.15.37:0/2703769091 >> v1:172.21.15.146:6789/0 conn(0x7f34800a9ce0 legacy=0x7f3480145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:01.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.277+0000 7f3484970640 1 -- 172.21.15.37:0/2703769091 >> v1:172.21.15.37:6789/0 conn(0x7f34800b3cd0 legacy=0x7f348014d7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:01.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3484970640 10 monclient: _finish_hunting 0 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3484970640 1 monclient: found mon.noname-b 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3484970640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3484970640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3484970640 1 -- 172.21.15.37:0/2703769091 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3480145e80 con 0x7f34800b4820 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3484970640 10 monclient: _finish_auth 0 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3484970640 10 monclient: _check_auth_tickets 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3484970640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3486974640 5 monclient: authenticate success, global_id 4268 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3486974640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3484970640 1 -- 172.21.15.37:0/2703769091 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3474001690 con 0x7f34800b4820 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3484970640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3484970640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3484970640 10 monclient: dump: 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:01.278 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:01.279 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:01.279 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:01.279 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:01.279 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:01.279 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:01.279 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:01.279 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:01.279 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:01.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.278+0000 7f3486974640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:01.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.280+0000 7f3484970640 1 -- 172.21.15.37:0/2703769091 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3474001920 con 0x7f34800b4820 2024-03-29T11:36:01.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.280+0000 7f3484970640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:01.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.280+0000 7f3484970640 1 -- 172.21.15.37:0/2703769091 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3474001cb0 con 0x7f34800b4820 2024-03-29T11:36:01.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.280+0000 7f3484970640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:01.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.280+0000 7f3484970640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:01.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.280+0000 7f3484970640 10 monclient: dump: 2024-03-29T11:36:01.280 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:01.280 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:01.280 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:01.280 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:01.280 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:01.280 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:01.280 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:01.280 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:01.280 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:01.280 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:01.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.282+0000 7f3486974640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:01.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.282+0000 7f3486974640 4 set_mon_vals no callback set 2024-03-29T11:36:01.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.283+0000 7f3486974640 10 monclient: shutdown 2024-03-29T11:36:01.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.283+0000 7f3486974640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:01.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.283+0000 7f3486974640 1 -- 172.21.15.37:0/2703769091 >> v1:172.21.15.37:6790/0 conn(0x7f34800b4820 legacy=0x7f348013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.283+0000 7f3486974640 1 -- 172.21.15.37:0/2703769091 shutdown_connections 2024-03-29T11:36:01.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.283+0000 7f3486974640 1 -- 172.21.15.37:0/2703769091 >> 172.21.15.37:0/2703769091 conn(0x7f348006d070 msgr2=0x7f3480066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:01.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.284+0000 7f3486974640 1 -- 172.21.15.37:0/2703769091 shutdown_connections 2024-03-29T11:36:01.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.284+0000 7f3486974640 1 -- 172.21.15.37:0/2703769091 wait complete. 2024-03-29T11:36:01.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.285+0000 7f3486974640 10 monclient: build_initial_monmap 2024-03-29T11:36:01.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.285+0000 7f3486974640 10 monclient: monmap: 2024-03-29T11:36:01.284 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:01.284 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:01.284 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:36:01.284 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:36:01.284 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:01.284 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:01.285 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:01.285 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:36:01.285 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:36:01.285 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:01.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.285+0000 7f3486974640 1 librados: starting msgr at 2024-03-29T11:36:01.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.285+0000 7f3486974640 1 librados: starting objecter 2024-03-29T11:36:01.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.285+0000 7f3486974640 1 Processor -- start 2024-03-29T11:36:01.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.285+0000 7f3486974640 1 -- start start 2024-03-29T11:36:01.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.285+0000 7f3486974640 1 librados: setting wanted keys 2024-03-29T11:36:01.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.285+0000 7f3486974640 1 librados: calling monclient init 2024-03-29T11:36:01.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.285+0000 7f3486974640 10 monclient: init 2024-03-29T11:36:01.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.286+0000 7f3486974640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:01.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.286+0000 7f3486974640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:01.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.286+0000 7f3486974640 10 monclient(hunting): picked mon.noname-a con 0x7f34800a9ce0 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:01.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.286+0000 7f3486974640 10 monclient(hunting): picked mon.noname-c con 0x7f34800b3cd0 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:01.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.286+0000 7f3486974640 10 monclient(hunting): picked mon.noname-b con 0x7f34800b4820 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:01.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.286+0000 7f3486974640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34801d6ea0 con 0x7f34800a9ce0 2024-03-29T11:36:01.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.286+0000 7f3486974640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348013f680 con 0x7f34800b4820 2024-03-29T11:36:01.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.286+0000 7f3486974640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348006e7d0 con 0x7f34800b3cd0 2024-03-29T11:36:01.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.286+0000 7f3486974640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:01.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.286+0000 7f3486974640 10 monclient(hunting): authenticate will time out at 940.735546s 2024-03-29T11:36:01.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.290+0000 7f3485972640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f34800a9ce0 0x7f34801d7330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46117/0 (socket says 172.21.15.37:46117) 2024-03-29T11:36:01.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.290+0000 7f3485972640 1 -- 172.21.15.37:0/174833123 learned_addr learned my addr 172.21.15.37:0/174833123 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:01.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 client.?.objecter ms_handle_connect 0x7f34800a9ce0 2024-03-29T11:36:01.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:01.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 client.?.objecter ms_handle_connect 0x7f34800b4820 2024-03-29T11:36:01.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:01.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232633625 0 0) 0x7f34801d6ea0 con 0x7f34800a9ce0 2024-03-29T11:36:01.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:01.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:01.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:01.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:36:01.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3464004eb0 con 0x7f34800a9ce0 2024-03-29T11:36:01.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602111673 0 0) 0x7f348013f680 con 0x7f34800b4820 2024-03-29T11:36:01.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:01.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:01.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:01.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:36:01.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34801d6ea0 con 0x7f34800b4820 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3038404457 0 0) 0x7f3464004eb0 con 0x7f34800a9ce0 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f348013f680 con 0x7f34800a9ce0 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3474001650 con 0x7f34800a9ce0 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 monclient(hunting): dump: 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:01.298 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.294+0000 7f34727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.295+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2659174141 0 0) 0x7f348013f680 con 0x7f34800a9ce0 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.295+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 >> v1:172.21.15.37:6790/0 conn(0x7f34800b3cd0 legacy=0x7f348013e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.295+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 >> v1:172.21.15.146:6789/0 conn(0x7f34800b4820 legacy=0x7f348013ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.295+0000 7f34727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.295+0000 7f34727fc640 1 monclient: found mon.a 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.295+0000 7f34727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.295+0000 7f34727fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.295+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34801d3ed0 con 0x7f34800a9ce0 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.296+0000 7f34727fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.296+0000 7f34727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.296+0000 7f34727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.296+0000 7f3486974640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:36:01.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.296+0000 7f3486974640 10 monclient: _renew_subs 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.296+0000 7f3486974640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.296+0000 7f3486974640 1 -- 172.21.15.37:0/174833123 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34801e8390 con 0x7f34800a9ce0 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.296+0000 7f3486974640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.296+0000 7f3486974640 10 monclient: _renew_subs 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.296+0000 7f3486974640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.296+0000 7f3486974640 1 -- 172.21.15.37:0/174833123 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34801d7c20 con 0x7f34800a9ce0 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.296+0000 7f3486974640 1 librados: init done 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.297+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3474001270 con 0x7f34800a9ce0 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.298+0000 7f34727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.298+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3474001650 con 0x7f34800a9ce0 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.298+0000 7f34727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.298+0000 7f34727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.298+0000 7f34727fc640 10 monclient: dump: 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:01.300 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:01.301 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:01.301 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:01.301 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:01.301 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:01.301 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:01.301 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:01.301 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:01.301 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:01.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.299+0000 7f3486974640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:01.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.299+0000 7f3486974640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:01.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.299+0000 7f3486974640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:36:01.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.299+0000 7f3486974640 1 -- 172.21.15.37:0/174833123 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348004f730 con 0x7f34800a9ce0 2024-03-29T11:36:01.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.299+0000 7f3473fff640 4 set_mon_vals no callback set 2024-03-29T11:36:01.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.299+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f347401aa40 con 0x7f34800a9ce0 2024-03-29T11:36:01.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.300+0000 7f3485171640 10 monclient: get_auth_request con 0x7f346406e130 auth_method 0 2024-03-29T11:36:01.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.300+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f3474001470 con 0x7f34800a9ce0 2024-03-29T11:36:01.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.300+0000 7f34727fc640 10 client.4260.objecter ms_dispatch 0x7f3480000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:01.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.300+0000 7f34727fc640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:01.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.300+0000 7f34727fc640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:01.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.300+0000 7f34727fc640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-29T11:36:01.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.301+0000 7f34727fc640 10 client.4260.objecter ms_handle_connect 0x7f346406e130 2024-03-29T11:36:01.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.311+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3474050860 con 0x7f34800a9ce0 2024-03-29T11:36:01.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.311+0000 7f34727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:01.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.311+0000 7f34727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:01.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.641+0000 7f3486974640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:01.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.642+0000 7f3486974640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:01.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.642+0000 7f3486974640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:36:01.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.642+0000 7f3486974640 1 -- 172.21.15.37:0/174833123 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34801d4060 con 0x7f34800a9ce0 2024-03-29T11:36:01.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.645+0000 7f34727fc640 1 -- 172.21.15.37:0/174833123 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f3474050a00 con 0x7f34800a9ce0 2024-03-29T11:36:01.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.645+0000 7f34727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:01.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.645+0000 7f34727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:01.646 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:36:01.646 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:01.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.648+0000 7f3486974640 10 librados: watch_flush enter 2024-03-29T11:36:01.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.649+0000 7f3486974640 10 librados: watch_flush exit 2024-03-29T11:36:01.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.649+0000 7f3486974640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-29T11:36:01.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.649+0000 7f3486974640 10 client.4260.objecter successfully canceled tick 2024-03-29T11:36:01.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.649+0000 7f3486974640 1 -- 172.21.15.37:0/174833123 >> v1:172.21.15.146:6816/2899149139 conn(0x7f346406e130 legacy=0x7f34640705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.649+0000 7f3486974640 10 monclient: shutdown 2024-03-29T11:36:01.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.649+0000 7f3486974640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:01.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.649+0000 7f3486974640 1 -- 172.21.15.37:0/174833123 >> v1:172.21.15.37:6789/0 conn(0x7f34800a9ce0 legacy=0x7f34801d7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.652+0000 7f3486974640 1 -- 172.21.15.37:0/174833123 shutdown_connections 2024-03-29T11:36:01.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.652+0000 7f3486974640 1 -- 172.21.15.37:0/174833123 >> 172.21.15.37:0/174833123 conn(0x7f348006d070 msgr2=0x7f3480066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:01.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.653+0000 7f3486974640 1 -- 172.21.15.37:0/174833123 shutdown_connections 2024-03-29T11:36:01.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.653+0000 7f3486974640 1 -- 172.21.15.37:0/174833123 wait complete. 2024-03-29T11:36:01.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:01.653+0000 7f3486974640 1 librados: shutdown 2024-03-29T11:36:01.670 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:08.477 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:08.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.635+0000 7f66bb590640 10 monclient: get_monmap_and_config 2024-03-29T11:36:08.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.635+0000 7f66bb590640 10 monclient: build_initial_monmap 2024-03-29T11:36:08.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.635+0000 7f66bb590640 10 monclient: monmap: 2024-03-29T11:36:08.634 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:08.634 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:08.634 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:36:08.636305+0000 2024-03-29T11:36:08.634 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:36:08.636305+0000 2024-03-29T11:36:08.634 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:08.634 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:08.634 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:08.634 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:36:08.634 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:36:08.635 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:08.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.637+0000 7f66bb590640 1 Processor -- start 2024-03-29T11:36:08.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.637+0000 7f66bb590640 1 -- start start 2024-03-29T11:36:08.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.637+0000 7f66bb590640 10 monclient: init 2024-03-29T11:36:08.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.638+0000 7f66bb590640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:08.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.638+0000 7f66bb590640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:08.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.638+0000 7f66bb590640 10 monclient(hunting): picked mon.noname-b con 0x7f66b40b0860 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:08.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.638+0000 7f66bb590640 10 monclient(hunting): picked mon.noname-a con 0x7f66b413ef30 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:08.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.638+0000 7f66bb590640 10 monclient(hunting): picked mon.noname-c con 0x7f66b40af8e0 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:08.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.638+0000 7f66bb590640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b40b0c40 con 0x7f66b413ef30 2024-03-29T11:36:08.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.638+0000 7f66bb590640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b40aed10 con 0x7f66b40af8e0 2024-03-29T11:36:08.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.638+0000 7f66bb590640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b40afd50 con 0x7f66b40b0860 2024-03-29T11:36:08.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.638+0000 7f66bb590640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:08.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.638+0000 7f66bb590640 10 monclient(hunting): authenticate will time out at 948.087599s 2024-03-29T11:36:08.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.643+0000 7f66b9b2d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f66b40af8e0 0x7f66b4149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:40997/0 (socket says 172.21.15.37:40997) 2024-03-29T11:36:08.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.643+0000 7f66b9b2d640 1 -- 172.21.15.37:0/2012957965 learned_addr learned my addr 172.21.15.37:0/2012957965 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:08.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.643+0000 7f66b8b2b640 1 --1- 172.21.15.37:0/2012957965 >> v1:172.21.15.37:6789/0 conn(0x7f66b413ef30 0x7f66b40af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:45461/0 (socket says 172.21.15.37:45461) 2024-03-29T11:36:08.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.644+0000 7f66abfff640 1 -- 172.21.15.37:0/2012957965 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345764332 0 0) 0x7f66b40aed10 con 0x7f66b40af8e0 2024-03-29T11:36:08.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.644+0000 7f66abfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:08.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.644+0000 7f66abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:08.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.644+0000 7f66abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:08.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.644+0000 7f66abfff640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:36:08.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.644+0000 7f66abfff640 1 -- 172.21.15.37:0/2012957965 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a0003520 con 0x7f66b40af8e0 2024-03-29T11:36:08.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.644+0000 7f66abfff640 1 -- 172.21.15.37:0/2012957965 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523848239 0 0) 0x7f66b40b0c40 con 0x7f66b413ef30 2024-03-29T11:36:08.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.644+0000 7f66abfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:08.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.644+0000 7f66abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:08.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.644+0000 7f66abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:08.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.644+0000 7f66abfff640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:36:08.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.644+0000 7f66abfff640 1 -- 172.21.15.37:0/2012957965 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b40aed10 con 0x7f66b413ef30 2024-03-29T11:36:08.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.645+0000 7f66abfff640 1 -- 172.21.15.37:0/2012957965 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3355979994 0 0) 0x7f66a0003520 con 0x7f66b40af8e0 2024-03-29T11:36:08.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.645+0000 7f66abfff640 1 -- 172.21.15.37:0/2012957965 >> v1:172.21.15.37:6790/0 conn(0x7f66b40b0860 legacy=0x7f66b413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.645+0000 7f66abfff640 1 -- 172.21.15.37:0/2012957965 >> v1:172.21.15.37:6789/0 conn(0x7f66b413ef30 legacy=0x7f66b40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:08.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 1 monclient: found mon.noname-c 2024-03-29T11:36:08.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:08.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:36:08.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 1 -- 172.21.15.37:0/2012957965 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b40ab1b0 con 0x7f66b40af8e0 2024-03-29T11:36:08.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 10 monclient: _finish_auth 0 2024-03-29T11:36:08.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:08.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:08.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 1 -- 172.21.15.37:0/2012957965 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f66b0004c90 con 0x7f66b40af8e0 2024-03-29T11:36:08.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66bb590640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:36:08.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66bb590640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:08.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:08.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:08.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 10 monclient: dump: 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 1 -- 172.21.15.37:0/2012957965 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66b0004eb0 con 0x7f66b40af8e0 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66bb590640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66abfff640 1 -- 172.21.15.37:0/2012957965 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f66b0001410 con 0x7f66b40af8e0 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66bb590640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:08.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.646+0000 7f66bb590640 4 set_mon_vals no callback set 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.647+0000 7f66abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.647+0000 7f66abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.647+0000 7f66abfff640 10 monclient: dump: 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.647+0000 7f66bb590640 10 monclient: shutdown 2024-03-29T11:36:08.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.647+0000 7f66bb590640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:08.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.647+0000 7f66bb590640 1 -- 172.21.15.37:0/2012957965 >> v1:172.21.15.146:6789/0 conn(0x7f66b40af8e0 legacy=0x7f66b4149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.648+0000 7f66bb590640 1 -- 172.21.15.37:0/2012957965 shutdown_connections 2024-03-29T11:36:08.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.648+0000 7f66bb590640 1 -- 172.21.15.37:0/2012957965 >> 172.21.15.37:0/2012957965 conn(0x7f66b406b660 msgr2=0x7f66b406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:08.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.654+0000 7f66bb590640 1 -- 172.21.15.37:0/2012957965 shutdown_connections 2024-03-29T11:36:08.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.654+0000 7f66bb590640 1 -- 172.21.15.37:0/2012957965 wait complete. 2024-03-29T11:36:08.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.656+0000 7f66bb590640 10 monclient: build_initial_monmap 2024-03-29T11:36:08.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.656+0000 7f66bb590640 10 monclient: monmap: 2024-03-29T11:36:08.654 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:08.654 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:08.654 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:36:08.654 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:36:08.655 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:08.655 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:08.655 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:08.655 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:36:08.655 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:36:08.655 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:08.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.656+0000 7f66bb590640 1 librados: starting msgr at 2024-03-29T11:36:08.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.656+0000 7f66bb590640 1 librados: starting objecter 2024-03-29T11:36:08.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.657+0000 7f66bb590640 1 Processor -- start 2024-03-29T11:36:08.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.657+0000 7f66bb590640 1 -- start start 2024-03-29T11:36:08.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.657+0000 7f66bb590640 1 librados: setting wanted keys 2024-03-29T11:36:08.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.657+0000 7f66bb590640 1 librados: calling monclient init 2024-03-29T11:36:08.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.657+0000 7f66bb590640 10 monclient: init 2024-03-29T11:36:08.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.658+0000 7f66bb590640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:08.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.658+0000 7f66bb590640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:08.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.662+0000 7f66bb590640 10 monclient(hunting): picked mon.noname-b con 0x7f66b40af8e0 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:08.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.662+0000 7f66b932c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f66b40af8e0 0x7f66b4136c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:41109/0 (socket says 172.21.15.37:41109) 2024-03-29T11:36:08.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.662+0000 7f66b932c640 1 -- 172.21.15.37:0/2571982706 learned_addr learned my addr 172.21.15.37:0/2571982706 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:08.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.662+0000 7f66bb590640 10 monclient(hunting): picked mon.noname-c con 0x7f66b40b0860 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:08.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.664+0000 7f66bb590640 10 monclient(hunting): picked mon.noname-a con 0x7f66b413ef30 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:08.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.664+0000 7f66bb590640 1 -- 172.21.15.37:0/2571982706 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b41367e0 con 0x7f66b413ef30 2024-03-29T11:36:08.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.664+0000 7f66bb590640 1 -- 172.21.15.37:0/2571982706 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b41364b0 con 0x7f66b40af8e0 2024-03-29T11:36:08.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.664+0000 7f66bb590640 1 -- 172.21.15.37:0/2571982706 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b4137380 con 0x7f66b40b0860 2024-03-29T11:36:08.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.664+0000 7f66bb590640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:08.663 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.664+0000 7f66bb590640 10 monclient(hunting): authenticate will time out at 948.113082s 2024-03-29T11:36:08.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.662+0000 7f66a9ffb640 10 client.?.objecter ms_handle_connect 0x7f66b40af8e0 2024-03-29T11:36:08.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.665+0000 7f66a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:08.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.665+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283074292 0 0) 0x7f66b41364b0 con 0x7f66b40af8e0 2024-03-29T11:36:08.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.665+0000 7f66a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:08.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.665+0000 7f66a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:08.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.665+0000 7f66a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:08.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.665+0000 7f66a9ffb640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:36:08.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.665+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6698004eb0 con 0x7f66b40af8e0 2024-03-29T11:36:08.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.666+0000 7f66a9ffb640 10 client.?.objecter ms_handle_connect 0x7f66b40b0860 2024-03-29T11:36:08.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.666+0000 7f66a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.666+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035700346 0 0) 0x7f66b4137380 con 0x7f66b40b0860 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.666+0000 7f66a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.666+0000 7f66a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.666+0000 7f66a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.666+0000 7f66a9ffb640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.666+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b41364b0 con 0x7f66b40b0860 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1210542217 0 0) 0x7f6698004eb0 con 0x7f66b40af8e0 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66b4137380 con 0x7f66b40af8e0 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f66ac001db0 con 0x7f66b40af8e0 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:08.666 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 613420612 0 0) 0x7f66b41364b0 con 0x7f66b40b0860 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6698004eb0 con 0x7f66b40b0860 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f66a4001850 con 0x7f66b40b0860 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:08.667 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:08.668 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:08.668 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:08.668 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:08.668 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:08.668 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:08.668 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:08.668 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:08.668 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:08.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 899582801 0 0) 0x7f6698004eb0 con 0x7f66b40b0860 2024-03-29T11:36:08.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 >> v1:172.21.15.146:6789/0 conn(0x7f66b40af8e0 legacy=0x7f66b4136c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 >> v1:172.21.15.37:6789/0 conn(0x7f66b413ef30 legacy=0x7f66b41d6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:08.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 1 monclient: found mon.c 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.667+0000 7f66a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.668+0000 7f66a9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.668+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b4132e90 con 0x7f66b40b0860 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.668+0000 7f66a9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.668+0000 7f66a9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.668+0000 7f66a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.668+0000 7f66bb590640 5 monclient: authenticate success, global_id 4274 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.668+0000 7f66bb590640 10 monclient: _renew_subs 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.668+0000 7f66bb590640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.668+0000 7f66bb590640 1 -- 172.21.15.37:0/2571982706 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b4133160 con 0x7f66b40b0860 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.668+0000 7f66bb590640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.668+0000 7f66bb590640 10 monclient: _renew_subs 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.668+0000 7f66bb590640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.668+0000 7f66bb590640 1 -- 172.21.15.37:0/2571982706 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b41c7160 con 0x7f66b40b0860 2024-03-29T11:36:08.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.668+0000 7f66bb590640 1 librados: init done 2024-03-29T11:36:08.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.671+0000 7f66bb590640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:08.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.671+0000 7f66bb590640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:08.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.671+0000 7f66bb590640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:08.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.671+0000 7f66bb590640 1 -- 172.21.15.37:0/2571982706 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b41385a0 con 0x7f66b40b0860 2024-03-29T11:36:08.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.672+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66a4001d40 con 0x7f66b40b0860 2024-03-29T11:36:08.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.672+0000 7f66a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:08.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.672+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f66a4004980 con 0x7f66b40b0860 2024-03-29T11:36:08.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.672+0000 7f66a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:08.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.672+0000 7f66a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:08.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.672+0000 7f66a9ffb640 10 monclient: dump: 2024-03-29T11:36:08.671 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:08.671 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:08.671 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:08.671 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:08.672 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:08.672 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:08.672 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:08.672 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:08.672 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:08.672 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:08.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.673+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f66a401a950 con 0x7f66b40b0860 2024-03-29T11:36:08.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.674+0000 7f66b932c640 10 monclient: get_auth_request con 0x7f669806de80 auth_method 0 2024-03-29T11:36:08.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.674+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f66a40849d0 con 0x7f66b40b0860 2024-03-29T11:36:08.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.674+0000 7f66ab7fe640 4 set_mon_vals no callback set 2024-03-29T11:36:08.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.674+0000 7f66a9ffb640 10 client.4274.objecter ms_dispatch 0x7f66b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:08.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.674+0000 7f66a9ffb640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:08.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.674+0000 7f66a9ffb640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:08.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.674+0000 7f66a9ffb640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T11:36:08.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.674+0000 7f66a9ffb640 10 client.4274.objecter ms_handle_connect 0x7f669806de80 2024-03-29T11:36:08.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.682+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f66a40502b0 con 0x7f66b40b0860 2024-03-29T11:36:08.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.682+0000 7f66a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:08.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:08.682+0000 7f66a9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:09.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.037+0000 7f66bb590640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:09.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.037+0000 7f66bb590640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:09.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.037+0000 7f66bb590640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:09.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.037+0000 7f66bb590640 1 -- 172.21.15.37:0/2571982706 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66b4136990 con 0x7f66b40b0860 2024-03-29T11:36:09.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.039+0000 7f66a9ffb640 1 -- 172.21.15.37:0/2571982706 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f66b4136990 con 0x7f66b40b0860 2024-03-29T11:36:09.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.039+0000 7f66a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:09.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.039+0000 7f66a9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:09.038 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:36:09.038 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:09.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.042+0000 7f66bb590640 10 librados: watch_flush enter 2024-03-29T11:36:09.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.042+0000 7f66bb590640 10 librados: watch_flush exit 2024-03-29T11:36:09.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.042+0000 7f66bb590640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T11:36:09.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.042+0000 7f66bb590640 10 client.4274.objecter successfully canceled tick 2024-03-29T11:36:09.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.043+0000 7f66bb590640 1 -- 172.21.15.37:0/2571982706 >> v1:172.21.15.146:6816/2899149139 conn(0x7f669806de80 legacy=0x7f6698070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.043+0000 7f66bb590640 10 monclient: shutdown 2024-03-29T11:36:09.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.043+0000 7f66bb590640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:09.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.043+0000 7f66bb590640 1 -- 172.21.15.37:0/2571982706 >> v1:172.21.15.37:6790/0 conn(0x7f66b40b0860 legacy=0x7f66b4132780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.043+0000 7f66bb590640 1 -- 172.21.15.37:0/2571982706 shutdown_connections 2024-03-29T11:36:09.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.043+0000 7f66bb590640 1 -- 172.21.15.37:0/2571982706 >> 172.21.15.37:0/2571982706 conn(0x7f66b406b660 msgr2=0x7f66b40adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:09.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.043+0000 7f66bb590640 1 -- 172.21.15.37:0/2571982706 shutdown_connections 2024-03-29T11:36:09.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.043+0000 7f66bb590640 1 -- 172.21.15.37:0/2571982706 wait complete. 2024-03-29T11:36:09.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:09.043+0000 7f66bb590640 1 librados: shutdown 2024-03-29T11:36:09.056 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:15.862 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:16.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.011+0000 7f3504856640 10 monclient: get_monmap_and_config 2024-03-29T11:36:16.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.011+0000 7f3504856640 10 monclient: build_initial_monmap 2024-03-29T11:36:16.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.011+0000 7f3504856640 10 monclient: monmap: 2024-03-29T11:36:16.011 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:16.011 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:16.011 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:36:16.013021+0000 2024-03-29T11:36:16.011 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:36:16.013021+0000 2024-03-29T11:36:16.011 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:16.011 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:16.011 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:16.012 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:36:16.012 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:36:16.012 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:16.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.012+0000 7f3504856640 1 Processor -- start 2024-03-29T11:36:16.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.012+0000 7f3504856640 1 -- start start 2024-03-29T11:36:16.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.012+0000 7f3504856640 10 monclient: init 2024-03-29T11:36:16.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.013+0000 7f3504856640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:16.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.013+0000 7f3504856640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:16.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.013+0000 7f3504856640 10 monclient(hunting): picked mon.noname-a con 0x7f34f8006eb0 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:16.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.013+0000 7f3504856640 10 monclient(hunting): picked mon.noname-b con 0x7f34f8038f10 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:16.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.013+0000 7f3504856640 10 monclient(hunting): picked mon.noname-c con 0x7f34f8036690 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:16.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.013+0000 7f3504856640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f8033c90 con 0x7f34f8006eb0 2024-03-29T11:36:16.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.013+0000 7f3504856640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f80389f0 con 0x7f34f8036690 2024-03-29T11:36:16.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.013+0000 7f3504856640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f8030890 con 0x7f34f8038f10 2024-03-29T11:36:16.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.013+0000 7f3504856640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:16.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.013+0000 7f3504856640 10 monclient(hunting): authenticate will time out at 955.463424s 2024-03-29T11:36:16.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.013+0000 7f34fe59c640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f34f8038f10 0x7f34f8039330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:36500/0 (socket says 172.21.15.37:36500) 2024-03-29T11:36:16.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fe59c640 1 -- 172.21.15.37:0/1478253752 learned_addr learned my addr 172.21.15.37:0/1478253752 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:16.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 1 -- 172.21.15.37:0/1478253752 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902238453 0 0) 0x7f34f80389f0 con 0x7f34f8036690 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 1 -- 172.21.15.37:0/1478253752 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34e8003520 con 0x7f34f8036690 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 1 -- 172.21.15.37:0/1478253752 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212706997 0 0) 0x7f34f8033c90 con 0x7f34f8006eb0 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 1 -- 172.21.15.37:0/1478253752 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f80389f0 con 0x7f34f8006eb0 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 1 -- 172.21.15.37:0/1478253752 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3310847037 0 0) 0x7f34e8003520 con 0x7f34f8036690 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 1 -- 172.21.15.37:0/1478253752 >> v1:172.21.15.37:6790/0 conn(0x7f34f8038f10 legacy=0x7f34f8039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 1 -- 172.21.15.37:0/1478253752 >> v1:172.21.15.37:6789/0 conn(0x7f34f8006eb0 legacy=0x7f34f8031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 1 monclient: found mon.noname-c 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 1 -- 172.21.15.37:0/1478253752 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f80cbc40 con 0x7f34f8036690 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient: _finish_auth 0 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f3504856640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f3504856640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 1 -- 172.21.15.37:0/1478253752 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f350004f730 con 0x7f34f8036690 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f34fdd9b640 10 monclient: dump: 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:16.016 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:16.017 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:16.017 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:16.017 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.017 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:16.017 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:16.017 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:16.017 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:16.017 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:16.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.014+0000 7f3504856640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:16.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.015+0000 7f34fdd9b640 1 -- 172.21.15.37:0/1478253752 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3500075f10 con 0x7f34f8036690 2024-03-29T11:36:16.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.015+0000 7f34fdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:16.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.015+0000 7f3504856640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:16.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.015+0000 7f3504856640 4 set_mon_vals no callback set 2024-03-29T11:36:16.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.016+0000 7f3504856640 10 monclient: shutdown 2024-03-29T11:36:16.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.016+0000 7f3504856640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:16.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.016+0000 7f3504856640 1 -- 172.21.15.37:0/1478253752 >> v1:172.21.15.146:6789/0 conn(0x7f34f8036690 legacy=0x7f34f8036b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.016+0000 7f34fdd9b640 1 -- 172.21.15.37:0/1478253752 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f35000758b0 con 0x7f34f8036690 2024-03-29T11:36:16.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.016+0000 7f34fdd9b640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:36:16.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.016+0000 7f3504856640 1 -- 172.21.15.37:0/1478253752 shutdown_connections 2024-03-29T11:36:16.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.016+0000 7f3504856640 1 -- 172.21.15.37:0/1478253752 >> 172.21.15.37:0/1478253752 conn(0x7f34f8005c20 msgr2=0x7f34f8006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:16.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.016+0000 7f3504856640 1 -- 172.21.15.37:0/1478253752 shutdown_connections 2024-03-29T11:36:16.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.016+0000 7f3504856640 1 -- 172.21.15.37:0/1478253752 wait complete. 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.017+0000 7f3504856640 10 monclient: build_initial_monmap 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.017+0000 7f3504856640 10 monclient: monmap: 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.018+0000 7f3504856640 1 librados: starting msgr at 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.018+0000 7f3504856640 1 librados: starting objecter 2024-03-29T11:36:16.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.018+0000 7f3504856640 1 Processor -- start 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.018+0000 7f3504856640 1 -- start start 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.018+0000 7f3504856640 1 librados: setting wanted keys 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.018+0000 7f3504856640 1 librados: calling monclient init 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.018+0000 7f3504856640 10 monclient: init 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.019+0000 7f3504856640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.019+0000 7f3504856640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.019+0000 7f3504856640 10 monclient(hunting): picked mon.noname-b con 0x7f34f8006eb0 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.019+0000 7f3504856640 10 monclient(hunting): picked mon.noname-a con 0x7f34f8036690 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.019+0000 7f3504856640 10 monclient(hunting): picked mon.noname-c con 0x7f34f8038f10 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.019+0000 7f3504856640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f800ba50 con 0x7f34f8036690 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.019+0000 7f3504856640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f8165760 con 0x7f34f8006eb0 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.019+0000 7f3504856640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f800a960 con 0x7f34f8038f10 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.019+0000 7f3504856640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.019+0000 7f3504856640 10 monclient(hunting): authenticate will time out at 955.469552s 2024-03-29T11:36:16.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.019+0000 7f34fed9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f34f8006eb0 0x7f34f800b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:54556/0 (socket says 172.21.15.37:54556) 2024-03-29T11:36:16.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.019+0000 7f34fed9d640 1 -- 172.21.15.37:0/2168794255 learned_addr learned my addr 172.21.15.37:0/2168794255 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:16.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.020+0000 7f34e77fe640 10 client.?.objecter ms_handle_connect 0x7f34f8006eb0 2024-03-29T11:36:16.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.020+0000 7f34e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:16.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.020+0000 7f34e77fe640 10 client.?.objecter ms_handle_connect 0x7f34f8036690 2024-03-29T11:36:16.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.020+0000 7f34e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:16.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.020+0000 7f34e77fe640 10 client.?.objecter ms_handle_connect 0x7f34f8038f10 2024-03-29T11:36:16.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.020+0000 7f34e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:16.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.020+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786190816 0 0) 0x7f34f8165760 con 0x7f34f8006eb0 2024-03-29T11:36:16.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.020+0000 7f34e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.020+0000 7f34e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.020+0000 7f34e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.020+0000 7f34e77fe640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:36:16.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.020+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ec004eb0 con 0x7f34f8006eb0 2024-03-29T11:36:16.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297303111 0 0) 0x7f34f800a960 con 0x7f34f8038f10 2024-03-29T11:36:16.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f8165760 con 0x7f34f8038f10 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4434699 0 0) 0x7f34f800ba50 con 0x7f34f8036690 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f800a960 con 0x7f34f8036690 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3489514795 0 0) 0x7f34ec004eb0 con 0x7f34f8006eb0 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34f800ba50 con 0x7f34f8006eb0 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f34f0001a10 con 0x7f34f8006eb0 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): dump: 2024-03-29T11:36:16.028 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:16.029 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:16.029 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:16.029 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:16.029 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.029 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:16.029 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:16.029 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:16.029 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:16.029 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:16.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:16.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 742216720 0 0) 0x7f34f8165760 con 0x7f34f8038f10 2024-03-29T11:36:16.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34ec004eb0 con 0x7f34f8038f10 2024-03-29T11:36:16.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f350004f400 con 0x7f34f8038f10 2024-03-29T11:36:16.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:16.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.021+0000 7f34e77fe640 10 monclient(hunting): dump: 2024-03-29T11:36:16.030 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:16.030 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:16.030 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:16.030 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:16.030 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.030 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:16.030 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:16.030 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:16.030 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:16.030 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:16.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2303153439 0 0) 0x7f34f800ba50 con 0x7f34f8006eb0 2024-03-29T11:36:16.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 >> v1:172.21.15.37:6790/0 conn(0x7f34f8038f10 legacy=0x7f34f8165050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 >> v1:172.21.15.37:6789/0 conn(0x7f34f8036690 legacy=0x7f34f8164940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f34e77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f34e77fe640 1 monclient: found mon.b 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f34e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f34e77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f800ac40 con 0x7f34f8006eb0 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f34e77fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f34e77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f34e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f3504856640 5 monclient: authenticate success, global_id 4276 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f3504856640 10 monclient: _renew_subs 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f3504856640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f3504856640 1 -- 172.21.15.37:0/2168794255 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f800af20 con 0x7f34f8006eb0 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.022+0000 7f3504856640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.023+0000 7f3504856640 10 monclient: _renew_subs 2024-03-29T11:36:16.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.023+0000 7f3504856640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.023+0000 7f3504856640 1 -- 172.21.15.37:0/2168794255 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f802f5c0 con 0x7f34f8006eb0 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.023+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34f0001c50 con 0x7f34f8006eb0 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.023+0000 7f3504856640 1 librados: init done 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.023+0000 7f34e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.023+0000 7f34fd59a640 4 set_mon_vals no callback set 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.023+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f34f0003a60 con 0x7f34f8006eb0 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.023+0000 7f34e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.023+0000 7f34e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.023+0000 7f34e77fe640 10 monclient: dump: 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.032 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.024+0000 7f3504856640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.024+0000 7f3504856640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.024+0000 7f3504856640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.024+0000 7f3504856640 1 -- 172.21.15.37:0/2168794255 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f8036aa0 con 0x7f34f8006eb0 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.025+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f34f0003bc0 con 0x7f34f8006eb0 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.026+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f34f0084ad0 con 0x7f34f8006eb0 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.026+0000 7f34e77fe640 10 client.4276.objecter ms_dispatch 0x7f3500000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.026+0000 7f34fe59c640 10 monclient: get_auth_request con 0x7f34ec06dee0 auth_method 0 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.026+0000 7f34e77fe640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.026+0000 7f34e77fe640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.026+0000 7f34e77fe640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-29T11:36:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.026+0000 7f34e77fe640 10 client.4276.objecter ms_handle_connect 0x7f34ec06dee0 2024-03-29T11:36:16.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.032+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34f0086e60 con 0x7f34f8006eb0 2024-03-29T11:36:16.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.032+0000 7f34e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:16.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.032+0000 7f34e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:16.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.412+0000 7f3504856640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:16.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.412+0000 7f3504856640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:16.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.412+0000 7f3504856640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:16.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.412+0000 7f3504856640 1 -- 172.21.15.37:0/2168794255 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34f8007b70 con 0x7f34f8006eb0 2024-03-29T11:36:16.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.414+0000 7f34e77fe640 1 -- 172.21.15.37:0/2168794255 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f34f00503b0 con 0x7f34f8006eb0 2024-03-29T11:36:16.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.414+0000 7f34e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:16.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.414+0000 7f34e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:16.414 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:36:16.414 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:16.418 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.417+0000 7f3504856640 10 librados: watch_flush enter 2024-03-29T11:36:16.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.417+0000 7f3504856640 10 librados: watch_flush exit 2024-03-29T11:36:16.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.417+0000 7f3504856640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-29T11:36:16.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.417+0000 7f3504856640 10 client.4276.objecter successfully canceled tick 2024-03-29T11:36:16.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.417+0000 7f3504856640 1 -- 172.21.15.37:0/2168794255 >> v1:172.21.15.146:6816/2899149139 conn(0x7f34ec06dee0 legacy=0x7f34ec0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.417+0000 7f3504856640 10 monclient: shutdown 2024-03-29T11:36:16.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.417+0000 7f3504856640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:16.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.417+0000 7f3504856640 1 -- 172.21.15.37:0/2168794255 >> v1:172.21.15.146:6789/0 conn(0x7f34f8006eb0 legacy=0x7f34f800b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.418+0000 7f3504856640 1 -- 172.21.15.37:0/2168794255 shutdown_connections 2024-03-29T11:36:16.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.418+0000 7f3504856640 1 -- 172.21.15.37:0/2168794255 >> 172.21.15.37:0/2168794255 conn(0x7f34f8005c20 msgr2=0x7f34f803ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:16.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.418+0000 7f3504856640 1 -- 172.21.15.37:0/2168794255 shutdown_connections 2024-03-29T11:36:16.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.418+0000 7f3504856640 1 -- 172.21.15.37:0/2168794255 wait complete. 2024-03-29T11:36:16.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:16.418+0000 7f3504856640 1 librados: shutdown 2024-03-29T11:36:16.433 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:23.239 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:23.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.384+0000 7fe287698640 10 monclient: get_monmap_and_config 2024-03-29T11:36:23.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.384+0000 7fe287698640 10 monclient: build_initial_monmap 2024-03-29T11:36:23.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.384+0000 7fe287698640 10 monclient: monmap: 2024-03-29T11:36:23.385 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:23.385 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:23.385 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:36:23.385973+0000 2024-03-29T11:36:23.385 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:36:23.385973+0000 2024-03-29T11:36:23.385 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:23.385 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:23.385 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:23.385 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:36:23.385 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:36:23.385 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:23.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.385+0000 7fe287698640 1 Processor -- start 2024-03-29T11:36:23.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.385+0000 7fe287698640 1 -- start start 2024-03-29T11:36:23.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.385+0000 7fe287698640 10 monclient: init 2024-03-29T11:36:23.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.386+0000 7fe287698640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:23.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.386+0000 7fe287698640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:23.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.386+0000 7fe287698640 10 monclient(hunting): picked mon.noname-c con 0x7fe2800a9ac0 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:23.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.386+0000 7fe287698640 10 monclient(hunting): picked mon.noname-a con 0x7fe28013ee90 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:23.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.387+0000 7fe287698640 10 monclient(hunting): picked mon.noname-b con 0x7fe2800af900 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:23.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.387+0000 7fe287698640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2800afd10 con 0x7fe28013ee90 2024-03-29T11:36:23.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.387+0000 7fe287698640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2800af6f0 con 0x7fe2800a9ac0 2024-03-29T11:36:23.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.387+0000 7fe287698640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2800aac80 con 0x7fe2800af900 2024-03-29T11:36:23.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.387+0000 7fe287698640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:23.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.387+0000 7fe287698640 10 monclient(hunting): authenticate will time out at 962.837211s 2024-03-29T11:36:23.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.388+0000 7fe285434640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe2800a9ac0 0x7fe2800adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:51710/0 (socket says 172.21.15.37:51710) 2024-03-29T11:36:23.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.388+0000 7fe285434640 1 -- 172.21.15.37:0/3967747951 learned_addr learned my addr 172.21.15.37:0/3967747951 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:23.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.388+0000 7fe27ffff640 1 -- 172.21.15.37:0/3967747951 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838353040 0 0) 0x7fe2800af6f0 con 0x7fe2800a9ac0 2024-03-29T11:36:23.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.388+0000 7fe27ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:23.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.388+0000 7fe27ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:23.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.388+0000 7fe27ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:23.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.388+0000 7fe27ffff640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:36:23.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.388+0000 7fe27ffff640 1 -- 172.21.15.37:0/3967747951 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe268003520 con 0x7fe2800a9ac0 2024-03-29T11:36:23.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.389+0000 7fe27ffff640 1 -- 172.21.15.37:0/3967747951 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 408012220 0 0) 0x7fe268003520 con 0x7fe2800a9ac0 2024-03-29T11:36:23.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.389+0000 7fe27ffff640 1 -- 172.21.15.37:0/3967747951 >> v1:172.21.15.37:6790/0 conn(0x7fe2800af900 legacy=0x7fe280149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:23.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.389+0000 7fe27ffff640 1 -- 172.21.15.37:0/3967747951 >> v1:172.21.15.37:6789/0 conn(0x7fe28013ee90 legacy=0x7fe28013b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:23.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:23.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 1 monclient: found mon.noname-c 2024-03-29T11:36:23.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:23.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:36:23.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 1 -- 172.21.15.37:0/3967747951 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2800aaf60 con 0x7fe2800a9ac0 2024-03-29T11:36:23.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 10 monclient: _finish_auth 0 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 1 -- 172.21.15.37:0/3967747951 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fe270001a10 con 0x7fe2800a9ac0 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 10 monclient: dump: 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:23.393 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:23.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe287698640 5 monclient: authenticate success, global_id 4279 2024-03-29T11:36:23.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe287698640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:23.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 1 -- 172.21.15.37:0/3967747951 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe270001ca0 con 0x7fe2800a9ac0 2024-03-29T11:36:23.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:23.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 1 -- 172.21.15.37:0/3967747951 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fe270002030 con 0x7fe2800a9ac0 2024-03-29T11:36:23.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:23.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:23.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.390+0000 7fe27ffff640 10 monclient: dump: 2024-03-29T11:36:23.394 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:23.394 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:23.394 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:23.394 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:23.394 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:23.394 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:23.395 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:23.395 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:23.395 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:23.395 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:23.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.391+0000 7fe287698640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:23.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.391+0000 7fe287698640 4 set_mon_vals no callback set 2024-03-29T11:36:23.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.391+0000 7fe287698640 10 monclient: shutdown 2024-03-29T11:36:23.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.391+0000 7fe287698640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:23.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.391+0000 7fe287698640 1 -- 172.21.15.37:0/3967747951 >> v1:172.21.15.146:6789/0 conn(0x7fe2800a9ac0 legacy=0x7fe2800adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.392+0000 7fe287698640 1 -- 172.21.15.37:0/3967747951 shutdown_connections 2024-03-29T11:36:23.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.392+0000 7fe287698640 1 -- 172.21.15.37:0/3967747951 >> 172.21.15.37:0/3967747951 conn(0x7fe2800b35e0 msgr2=0x7fe2800b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:23.396 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.393+0000 7fe287698640 1 -- 172.21.15.37:0/3967747951 shutdown_connections 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.393+0000 7fe287698640 1 -- 172.21.15.37:0/3967747951 wait complete. 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.394+0000 7fe287698640 10 monclient: build_initial_monmap 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.394+0000 7fe287698640 10 monclient: monmap: 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.394+0000 7fe287698640 1 librados: starting msgr at 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.394+0000 7fe287698640 1 librados: starting objecter 2024-03-29T11:36:23.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.394+0000 7fe287698640 1 Processor -- start 2024-03-29T11:36:23.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.394+0000 7fe287698640 1 -- start start 2024-03-29T11:36:23.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.394+0000 7fe287698640 1 librados: setting wanted keys 2024-03-29T11:36:23.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.394+0000 7fe287698640 1 librados: calling monclient init 2024-03-29T11:36:23.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.394+0000 7fe287698640 10 monclient: init 2024-03-29T11:36:23.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.395+0000 7fe287698640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:23.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.395+0000 7fe287698640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:23.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.396+0000 7fe287698640 10 monclient(hunting): picked mon.noname-c con 0x7fe2800a9ac0 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:23.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.396+0000 7fe287698640 10 monclient(hunting): picked mon.noname-a con 0x7fe2800af900 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:23.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.396+0000 7fe287698640 10 monclient(hunting): picked mon.noname-b con 0x7fe28013ee90 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:23.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.396+0000 7fe287698640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28012cff0 con 0x7fe2800af900 2024-03-29T11:36:23.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.396+0000 7fe287698640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28012cd10 con 0x7fe28013ee90 2024-03-29T11:36:23.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.396+0000 7fe287698640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28014e430 con 0x7fe2800a9ac0 2024-03-29T11:36:23.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.396+0000 7fe287698640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:23.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.396+0000 7fe287698640 10 monclient(hunting): authenticate will time out at 962.846376s 2024-03-29T11:36:23.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.397+0000 7fe285c35640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe28013ee90 0x7fe28014dd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:51938/0 (socket says 172.21.15.37:51938) 2024-03-29T11:36:23.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.397+0000 7fe285c35640 1 -- 172.21.15.37:0/1368423713 learned_addr learned my addr 172.21.15.37:0/1368423713 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:23.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 10 client.?.objecter ms_handle_connect 0x7fe28013ee90 2024-03-29T11:36:23.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:23.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 10 client.?.objecter ms_handle_connect 0x7fe2800a9ac0 2024-03-29T11:36:23.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:23.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468056981 0 0) 0x7fe28012cd10 con 0x7fe28013ee90 2024-03-29T11:36:23.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:23.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:23.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:23.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:36:23.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe26c004eb0 con 0x7fe28013ee90 2024-03-29T11:36:23.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621626692 0 0) 0x7fe28014e430 con 0x7fe2800a9ac0 2024-03-29T11:36:23.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:23.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:23.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:23.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:36:23.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.400+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe28012cd10 con 0x7fe2800a9ac0 2024-03-29T11:36:23.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 client.?.objecter ms_handle_connect 0x7fe2800af900 2024-03-29T11:36:23.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:23.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3919722755 0 0) 0x7fe28012cd10 con 0x7fe2800a9ac0 2024-03-29T11:36:23.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe28014e430 con 0x7fe2800a9ac0 2024-03-29T11:36:23.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3091323459 0 0) 0x7fe26c004eb0 con 0x7fe28013ee90 2024-03-29T11:36:23.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe28012cd10 con 0x7fe28013ee90 2024-03-29T11:36:23.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fe2780018e0 con 0x7fe28013ee90 2024-03-29T11:36:23.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient(hunting): dump: 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fe2700018d0 con 0x7fe2800a9ac0 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:23.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient(hunting): dump: 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757919369 0 0) 0x7fe28012cff0 con 0x7fe2800af900 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:23.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:36:23.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe26c004eb0 con 0x7fe2800af900 2024-03-29T11:36:23.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3451649516 0 0) 0x7fe28012cd10 con 0x7fe28013ee90 2024-03-29T11:36:23.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 >> v1:172.21.15.37:6790/0 conn(0x7fe2800a9ac0 legacy=0x7fe28014fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 >> v1:172.21.15.37:6789/0 conn(0x7fe2800af900 legacy=0x7fe28014d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:23.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 1 monclient: found mon.b 2024-03-29T11:36:23.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:23.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:23.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe280151050 con 0x7fe28013ee90 2024-03-29T11:36:23.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:23.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:23.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe27dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:23.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe287698640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:36:23.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe287698640 10 monclient: _renew_subs 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe287698640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.401+0000 7fe287698640 1 -- 172.21.15.37:0/1368423713 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe280151330 con 0x7fe28013ee90 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.402+0000 7fe287698640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.402+0000 7fe287698640 10 monclient: _renew_subs 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.402+0000 7fe287698640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.402+0000 7fe287698640 1 -- 172.21.15.37:0/1368423713 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe280150300 con 0x7fe28013ee90 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.402+0000 7fe287698640 1 librados: init done 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.402+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2780035f0 con 0x7fe28013ee90 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.402+0000 7fe27dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.402+0000 7fe27f7fe640 4 set_mon_vals no callback set 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.402+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fe2780039c0 con 0x7fe28013ee90 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.402+0000 7fe27dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.402+0000 7fe27dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.402+0000 7fe27dffb640 10 monclient: dump: 2024-03-29T11:36:23.411 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.403+0000 7fe287698640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.403+0000 7fe287698640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.403+0000 7fe287698640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.403+0000 7fe287698640 1 -- 172.21.15.37:0/1368423713 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe260005080 con 0x7fe28013ee90 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.405+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fe278003b20 con 0x7fe28013ee90 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.406+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fe2780849d0 con 0x7fe28013ee90 2024-03-29T11:36:23.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.406+0000 7fe27dffb640 10 client.4282.objecter ms_dispatch 0x7fe280000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:23.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.406+0000 7fe27dffb640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:23.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.406+0000 7fe27dffb640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:23.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.406+0000 7fe27dffb640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-29T11:36:23.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.406+0000 7fe285434640 10 monclient: get_auth_request con 0x7fe26c06df90 auth_method 0 2024-03-29T11:36:23.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.407+0000 7fe27dffb640 10 client.4282.objecter ms_handle_connect 0x7fe26c06df90 2024-03-29T11:36:23.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.412+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe2780502b0 con 0x7fe28013ee90 2024-03-29T11:36:23.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.412+0000 7fe27dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:23.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.412+0000 7fe27dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:23.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.762+0000 7fe287698640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:23.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.762+0000 7fe287698640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:23.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.762+0000 7fe287698640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:23.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.762+0000 7fe287698640 1 -- 172.21.15.37:0/1368423713 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe260005330 con 0x7fe28013ee90 2024-03-29T11:36:23.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.764+0000 7fe27dffb640 1 -- 172.21.15.37:0/1368423713 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fe2780521e0 con 0x7fe28013ee90 2024-03-29T11:36:23.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.764+0000 7fe27dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:23.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.764+0000 7fe27dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:23.766 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:36:23.766 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:23.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.769+0000 7fe287698640 10 librados: watch_flush enter 2024-03-29T11:36:23.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.769+0000 7fe287698640 10 librados: watch_flush exit 2024-03-29T11:36:23.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.769+0000 7fe287698640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-29T11:36:23.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.769+0000 7fe287698640 10 client.4282.objecter successfully canceled tick 2024-03-29T11:36:23.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.769+0000 7fe287698640 1 -- 172.21.15.37:0/1368423713 >> v1:172.21.15.146:6816/2899149139 conn(0x7fe26c06df90 legacy=0x7fe26c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.769+0000 7fe287698640 10 monclient: shutdown 2024-03-29T11:36:23.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.769+0000 7fe287698640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:23.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.769+0000 7fe287698640 1 -- 172.21.15.37:0/1368423713 >> v1:172.21.15.146:6789/0 conn(0x7fe28013ee90 legacy=0x7fe28014dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.769+0000 7fe287698640 1 -- 172.21.15.37:0/1368423713 shutdown_connections 2024-03-29T11:36:23.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.769+0000 7fe287698640 1 -- 172.21.15.37:0/1368423713 >> 172.21.15.37:0/1368423713 conn(0x7fe2800b35e0 msgr2=0x7fe28012fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:23.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.771+0000 7fe287698640 1 -- 172.21.15.37:0/1368423713 shutdown_connections 2024-03-29T11:36:23.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.772+0000 7fe287698640 1 -- 172.21.15.37:0/1368423713 wait complete. 2024-03-29T11:36:23.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:23.772+0000 7fe287698640 1 librados: shutdown 2024-03-29T11:36:23.789 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:30.596 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:30.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.746+0000 7f45540e4640 10 monclient: get_monmap_and_config 2024-03-29T11:36:30.746 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.746+0000 7f45540e4640 10 monclient: build_initial_monmap 2024-03-29T11:36:30.746 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.746+0000 7f45540e4640 10 monclient: monmap: 2024-03-29T11:36:30.746 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:30.746 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:30.746 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:36:30.747219+0000 2024-03-29T11:36:30.746 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:36:30.747219+0000 2024-03-29T11:36:30.746 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:30.747 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:30.747 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:30.747 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:36:30.747 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:36:30.747 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:30.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.747+0000 7f45540e4640 1 Processor -- start 2024-03-29T11:36:30.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.747+0000 7f45540e4640 1 -- start start 2024-03-29T11:36:30.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.747+0000 7f45540e4640 10 monclient: init 2024-03-29T11:36:30.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.748+0000 7f45540e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:30.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.748+0000 7f45540e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:30.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.751+0000 7f45540e4640 10 monclient(hunting): picked mon.noname-b con 0x7f454c1422b0 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:30.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.751+0000 7f45540e4640 10 monclient(hunting): picked mon.noname-a con 0x7f454c142690 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:30.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.751+0000 7f45540e4640 10 monclient(hunting): picked mon.noname-c con 0x7f454c0b17d0 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:30.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.751+0000 7f45540e4640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454c142c80 con 0x7f454c142690 2024-03-29T11:36:30.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.751+0000 7f45540e4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454c0aae40 con 0x7f454c0b17d0 2024-03-29T11:36:30.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.751+0000 7f45540e4640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454c13fbd0 con 0x7f454c1422b0 2024-03-29T11:36:30.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.751+0000 7f45540e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:30.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.751+0000 7f45540e4640 10 monclient(hunting): authenticate will time out at 970.200360s 2024-03-29T11:36:30.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.752+0000 7f4551e80640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f454c1422b0 0x7f454c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:55709/0 (socket says 172.21.15.37:55709) 2024-03-29T11:36:30.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.752+0000 7f4551e80640 1 -- 172.21.15.37:0/1294578518 learned_addr learned my addr 172.21.15.37:0/1294578518 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:30.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.752+0000 7f4550e7e640 1 -- 172.21.15.37:0/1294578518 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541979923 0 0) 0x7f454c13fbd0 con 0x7f454c1422b0 2024-03-29T11:36:30.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.752+0000 7f4550e7e640 10 monclient(hunting): _negotiate 2024-03-29T11:36:30.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.752+0000 7f4550e7e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:30.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.752+0000 7f4550e7e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:30.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.752+0000 7f4550e7e640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:36:30.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.752+0000 7f4550e7e640 1 -- 172.21.15.37:0/1294578518 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4538003520 con 0x7f454c1422b0 2024-03-29T11:36:30.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.753+0000 7f4550e7e640 1 -- 172.21.15.37:0/1294578518 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 188539580 0 0) 0x7f4538003520 con 0x7f454c1422b0 2024-03-29T11:36:30.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.753+0000 7f4550e7e640 1 -- 172.21.15.37:0/1294578518 >> v1:172.21.15.146:6789/0 conn(0x7f454c0b17d0 legacy=0x7f454c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:30.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.753+0000 7f4550e7e640 1 -- 172.21.15.37:0/1294578518 >> v1:172.21.15.37:6789/0 conn(0x7f454c142690 legacy=0x7f454c0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:30.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 10 monclient: _finish_hunting 0 2024-03-29T11:36:30.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 1 monclient: found mon.noname-b 2024-03-29T11:36:30.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:30.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:36:30.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 1 -- 172.21.15.37:0/1294578518 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c151940 con 0x7f454c1422b0 2024-03-29T11:36:30.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 10 monclient: _finish_auth 0 2024-03-29T11:36:30.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 10 monclient: _check_auth_tickets 2024-03-29T11:36:30.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:30.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 1 -- 172.21.15.37:0/1294578518 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4548001690 con 0x7f454c1422b0 2024-03-29T11:36:30.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:30.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:30.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 10 monclient: dump: 2024-03-29T11:36:30.756 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:30.756 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:30.756 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:30.756 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:30.756 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:30.756 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:30.756 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:30.757 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:30.757 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:30.757 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:30.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 1 -- 172.21.15.37:0/1294578518 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45480017f0 con 0x7f454c1422b0 2024-03-29T11:36:30.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:30.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 1 -- 172.21.15.37:0/1294578518 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4548001c40 con 0x7f454c1422b0 2024-03-29T11:36:30.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:30.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:30.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f4550e7e640 10 monclient: dump: 2024-03-29T11:36:30.757 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:30.757 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:30.758 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:30.758 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:30.758 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:30.758 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:30.758 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:30.758 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:30.758 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:30.758 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:30.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f45540e4640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:36:30.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f45540e4640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:30.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.755+0000 7f45540e4640 4 set_mon_vals no callback set 2024-03-29T11:36:30.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.756+0000 7f45540e4640 10 monclient: shutdown 2024-03-29T11:36:30.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.756+0000 7f45540e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:30.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.756+0000 7f45540e4640 1 -- 172.21.15.37:0/1294578518 >> v1:172.21.15.37:6790/0 conn(0x7f454c1422b0 legacy=0x7f454c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:30.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.757+0000 7f45540e4640 1 -- 172.21.15.37:0/1294578518 shutdown_connections 2024-03-29T11:36:30.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.757+0000 7f45540e4640 1 -- 172.21.15.37:0/1294578518 >> 172.21.15.37:0/1294578518 conn(0x7f454c066a00 msgr2=0x7f454c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:30.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.757+0000 7f45540e4640 1 -- 172.21.15.37:0/1294578518 shutdown_connections 2024-03-29T11:36:30.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.757+0000 7f45540e4640 1 -- 172.21.15.37:0/1294578518 wait complete. 2024-03-29T11:36:30.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.759+0000 7f45540e4640 10 monclient: build_initial_monmap 2024-03-29T11:36:30.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.759+0000 7f45540e4640 10 monclient: monmap: 2024-03-29T11:36:30.761 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:30.761 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:30.761 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:36:30.761 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:36:30.762 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:30.762 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:30.762 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:30.762 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:36:30.762 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:36:30.762 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:30.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.759+0000 7f45540e4640 1 librados: starting msgr at 2024-03-29T11:36:30.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.760+0000 7f45540e4640 1 librados: starting objecter 2024-03-29T11:36:30.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.760+0000 7f45540e4640 1 Processor -- start 2024-03-29T11:36:30.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.760+0000 7f45540e4640 1 -- start start 2024-03-29T11:36:30.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.760+0000 7f45540e4640 1 librados: setting wanted keys 2024-03-29T11:36:30.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.760+0000 7f45540e4640 1 librados: calling monclient init 2024-03-29T11:36:30.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.760+0000 7f45540e4640 10 monclient: init 2024-03-29T11:36:30.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.761+0000 7f45540e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:30.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.761+0000 7f45540e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:30.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.764+0000 7f4551e80640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f454c0b17d0 0x7f454c12e160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:56065/0 (socket says 172.21.15.37:56065) 2024-03-29T11:36:30.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.764+0000 7f4551e80640 1 -- 172.21.15.37:0/3482540120 learned_addr learned my addr 172.21.15.37:0/3482540120 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:30.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.764+0000 7f45540e4640 10 monclient(hunting): picked mon.noname-c con 0x7f454c0b17d0 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:30.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.764+0000 7f45540e4640 10 monclient(hunting): picked mon.noname-a con 0x7f454c1422b0 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:30.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.764+0000 7f45540e4640 10 monclient(hunting): picked mon.noname-b con 0x7f454c142690 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:30.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.764+0000 7f45427fc640 10 client.?.objecter ms_handle_connect 0x7f454c0b17d0 2024-03-29T11:36:30.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.764+0000 7f45540e4640 1 -- 172.21.15.37:0/3482540120 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454c14e510 con 0x7f454c1422b0 2024-03-29T11:36:30.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.764+0000 7f45540e4640 1 -- 172.21.15.37:0/3482540120 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454c0acbd0 con 0x7f454c142690 2024-03-29T11:36:30.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.764+0000 7f45427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:30.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.764+0000 7f45540e4640 1 -- 172.21.15.37:0/3482540120 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454c0aceb0 con 0x7f454c0b17d0 2024-03-29T11:36:30.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45540e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:30.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45540e4640 10 monclient(hunting): authenticate will time out at 970.213989s 2024-03-29T11:36:30.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45427fc640 10 client.?.objecter ms_handle_connect 0x7f454c142690 2024-03-29T11:36:30.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:30.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430780976 0 0) 0x7f454c0aceb0 con 0x7f454c0b17d0 2024-03-29T11:36:30.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:30.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:30.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:30.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45427fc640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:36:30.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4530004eb0 con 0x7f454c0b17d0 2024-03-29T11:36:30.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134738103 0 0) 0x7f454c0acbd0 con 0x7f454c142690 2024-03-29T11:36:30.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:30.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:30.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:30.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45427fc640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:36:30.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.765+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f454c0aceb0 con 0x7f454c142690 2024-03-29T11:36:30.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.766+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3072985529 0 0) 0x7f454c0aceb0 con 0x7f454c142690 2024-03-29T11:36:30.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.766+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f454c0acbd0 con 0x7f454c142690 2024-03-29T11:36:30.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.766+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4544001890 con 0x7f454c142690 2024-03-29T11:36:30.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.766+0000 7f45427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:30.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.766+0000 7f45427fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:30.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.766+0000 7f45427fc640 10 monclient(hunting): dump: 2024-03-29T11:36:30.771 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:30.771 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:30.771 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:30.771 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:30.771 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:30.771 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:30.771 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:30.771 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:30.771 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:30.771 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:30.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.766+0000 7f45427fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:30.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.766+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 945638818 0 0) 0x7f454c0acbd0 con 0x7f454c142690 2024-03-29T11:36:30.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.766+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 >> v1:172.21.15.37:6790/0 conn(0x7f454c0b17d0 legacy=0x7f454c12e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:30.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.766+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 >> v1:172.21.15.37:6789/0 conn(0x7f454c1422b0 legacy=0x7f454c14d6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:30.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.767+0000 7f45427fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:30.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.767+0000 7f45427fc640 1 monclient: found mon.b 2024-03-29T11:36:30.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.767+0000 7f45427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:30.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.767+0000 7f45427fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:30.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.767+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c12e870 con 0x7f454c142690 2024-03-29T11:36:30.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.767+0000 7f45427fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:30.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.767+0000 7f45427fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:30.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.767+0000 7f45427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:30.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.767+0000 7f45540e4640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:36:30.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.767+0000 7f45540e4640 10 monclient: _renew_subs 2024-03-29T11:36:30.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.767+0000 7f45540e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:30.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.767+0000 7f45540e4640 1 -- 172.21.15.37:0/3482540120 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f454c1507d0 con 0x7f454c142690 2024-03-29T11:36:30.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.768+0000 7f45540e4640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:30.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.768+0000 7f45540e4640 10 monclient: _renew_subs 2024-03-29T11:36:30.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.768+0000 7f45540e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:30.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.768+0000 7f45540e4640 1 -- 172.21.15.37:0/3482540120 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f454c0aba10 con 0x7f454c142690 2024-03-29T11:36:30.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.768+0000 7f45540e4640 1 librados: init done 2024-03-29T11:36:30.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.769+0000 7f45540e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:30.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.769+0000 7f45540e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:30.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.769+0000 7f45540e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:30.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.769+0000 7f45540e4640 1 -- 172.21.15.37:0/3482540120 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f454c075f10 con 0x7f454c142690 2024-03-29T11:36:30.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.769+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4544001da0 con 0x7f454c142690 2024-03-29T11:36:30.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.769+0000 7f45427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:30.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.769+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4544003c10 con 0x7f454c142690 2024-03-29T11:36:30.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.769+0000 7f45427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:30.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.769+0000 7f45427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:30.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.770+0000 7f45427fc640 10 monclient: dump: 2024-03-29T11:36:30.774 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:30.774 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:30.774 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:30.774 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:30.775 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:30.775 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:30.775 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:30.775 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:30.775 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:30.775 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:30.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.770+0000 7f4543fff640 4 set_mon_vals no callback set 2024-03-29T11:36:30.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.772+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f4544019be0 con 0x7f454c142690 2024-03-29T11:36:30.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.772+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f4544083a50 con 0x7f454c142690 2024-03-29T11:36:30.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.773+0000 7f4551e80640 10 monclient: get_auth_request con 0x7f453006dd30 auth_method 0 2024-03-29T11:36:30.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.773+0000 7f45427fc640 10 client.4285.objecter ms_dispatch 0x7f454c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:30.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.773+0000 7f45427fc640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:30.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.773+0000 7f45427fc640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:30.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.773+0000 7f45427fc640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-29T11:36:30.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.773+0000 7f45427fc640 10 client.4285.objecter ms_handle_connect 0x7f453006dd30 2024-03-29T11:36:30.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.779+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f454404f330 con 0x7f454c142690 2024-03-29T11:36:30.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.779+0000 7f45427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:30.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:30.779+0000 7f45427fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:31.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.107+0000 7f45540e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:31.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.107+0000 7f45540e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:31.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.107+0000 7f45540e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:36:31.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.107+0000 7f45540e4640 1 -- 172.21.15.37:0/3482540120 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f454c0b2270 con 0x7f454c142690 2024-03-29T11:36:31.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.108+0000 7f45427fc640 1 -- 172.21.15.37:0/3482540120 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f4544051260 con 0x7f454c142690 2024-03-29T11:36:31.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.108+0000 7f45427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:31.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.108+0000 7f45427fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:31.107 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:36:31.108 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:31.111 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.111+0000 7f45540e4640 10 librados: watch_flush enter 2024-03-29T11:36:31.111 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.112+0000 7f45540e4640 10 librados: watch_flush exit 2024-03-29T11:36:31.111 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.112+0000 7f45540e4640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-29T11:36:31.111 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.112+0000 7f45540e4640 10 client.4285.objecter successfully canceled tick 2024-03-29T11:36:31.111 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.112+0000 7f45540e4640 1 -- 172.21.15.37:0/3482540120 >> v1:172.21.15.146:6816/2899149139 conn(0x7f453006dd30 legacy=0x7f45300701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:31.111 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.112+0000 7f45540e4640 10 monclient: shutdown 2024-03-29T11:36:31.111 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.112+0000 7f45540e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:31.111 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.112+0000 7f45540e4640 1 -- 172.21.15.37:0/3482540120 >> v1:172.21.15.146:6789/0 conn(0x7f454c142690 legacy=0x7f454c14de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:31.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.113+0000 7f45540e4640 1 -- 172.21.15.37:0/3482540120 shutdown_connections 2024-03-29T11:36:31.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.113+0000 7f45540e4640 1 -- 172.21.15.37:0/3482540120 >> 172.21.15.37:0/3482540120 conn(0x7f454c066a00 msgr2=0x7f454c0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:31.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.115+0000 7f45540e4640 1 -- 172.21.15.37:0/3482540120 shutdown_connections 2024-03-29T11:36:31.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.115+0000 7f45540e4640 1 -- 172.21.15.37:0/3482540120 wait complete. 2024-03-29T11:36:31.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:31.115+0000 7f45540e4640 1 librados: shutdown 2024-03-29T11:36:31.131 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:37.937 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:38.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.083+0000 7f3575fd7640 10 monclient: get_monmap_and_config 2024-03-29T11:36:38.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.083+0000 7f3575fd7640 10 monclient: build_initial_monmap 2024-03-29T11:36:38.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.083+0000 7f3575fd7640 10 monclient: monmap: 2024-03-29T11:36:38.083 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:38.083 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:38.083 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:36:38.084638+0000 2024-03-29T11:36:38.083 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:36:38.084638+0000 2024-03-29T11:36:38.083 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:38.083 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:38.083 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:38.083 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:36:38.083 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:36:38.084 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:38.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.087+0000 7f3575fd7640 1 Processor -- start 2024-03-29T11:36:38.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.087+0000 7f3575fd7640 1 -- start start 2024-03-29T11:36:38.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.087+0000 7f3575fd7640 10 monclient: init 2024-03-29T11:36:38.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.088+0000 7f3575fd7640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:38.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.088+0000 7f3575fd7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:38.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.088+0000 7f3575fd7640 10 monclient(hunting): picked mon.noname-a con 0x7f35700b0860 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:38.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.088+0000 7f3575fd7640 10 monclient(hunting): picked mon.noname-c con 0x7f357013ef30 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:38.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.088+0000 7f3575fd7640 10 monclient(hunting): picked mon.noname-b con 0x7f35700af8e0 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:38.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.088+0000 7f3575fd7640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35700b0c40 con 0x7f35700b0860 2024-03-29T11:36:38.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.088+0000 7f3575fd7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35700aed10 con 0x7f357013ef30 2024-03-29T11:36:38.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.088+0000 7f3575fd7640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35700afd50 con 0x7f35700af8e0 2024-03-29T11:36:38.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.088+0000 7f3575fd7640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:38.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.088+0000 7f3575fd7640 10 monclient(hunting): authenticate will time out at 977.537278s 2024-03-29T11:36:38.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.093+0000 7f356ffff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f35700af8e0 0x7f3570149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:56273/0 (socket says 172.21.15.37:56273) 2024-03-29T11:36:38.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.093+0000 7f356ffff640 1 -- 172.21.15.37:0/1152442455 learned_addr learned my addr 172.21.15.37:0/1152442455 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:38.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.095+0000 7f356e7fc640 1 -- 172.21.15.37:0/1152442455 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951318194 0 0) 0x7f35700afd50 con 0x7f35700af8e0 2024-03-29T11:36:38.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.095+0000 7f356e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.095+0000 7f356e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.095+0000 7f356e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.095+0000 7f356e7fc640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:36:38.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.095+0000 7f356e7fc640 1 -- 172.21.15.37:0/1152442455 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3558003520 con 0x7f35700af8e0 2024-03-29T11:36:38.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.095+0000 7f356e7fc640 1 -- 172.21.15.37:0/1152442455 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192386891 0 0) 0x7f35700aed10 con 0x7f357013ef30 2024-03-29T11:36:38.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.095+0000 7f356e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.095+0000 7f356e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.095+0000 7f356e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.095+0000 7f356e7fc640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:36:38.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.095+0000 7f356e7fc640 1 -- 172.21.15.37:0/1152442455 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35700afd50 con 0x7f357013ef30 2024-03-29T11:36:38.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.095+0000 7f356e7fc640 1 -- 172.21.15.37:0/1152442455 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 166531804 0 0) 0x7f3558003520 con 0x7f35700af8e0 2024-03-29T11:36:38.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f356e7fc640 1 -- 172.21.15.37:0/1152442455 >> v1:172.21.15.146:6789/0 conn(0x7f357013ef30 legacy=0x7f35700af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f356e7fc640 1 -- 172.21.15.37:0/1152442455 >> v1:172.21.15.37:6789/0 conn(0x7f35700b0860 legacy=0x7f357013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:38.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f356e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:38.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f356e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:36:38.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f356e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f356e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f356e7fc640 1 -- 172.21.15.37:0/1152442455 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35700ab1b0 con 0x7f35700af8e0 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f356e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f356e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f356e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f3575fd7640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f3575fd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f356e7fc640 1 -- 172.21.15.37:0/1152442455 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3568005010 con 0x7f35700af8e0 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f356e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f356e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f356e7fc640 10 monclient: dump: 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:38.099 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:38.100 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.100 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:38.100 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:38.100 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:38.100 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:38.100 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:38.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.096+0000 7f3575fd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:38.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.097+0000 7f356e7fc640 1 -- 172.21.15.37:0/1152442455 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3568005230 con 0x7f35700af8e0 2024-03-29T11:36:38.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.097+0000 7f356e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:38.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.097+0000 7f3575fd7640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:38.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.097+0000 7f3575fd7640 4 set_mon_vals no callback set 2024-03-29T11:36:38.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.098+0000 7f3575fd7640 10 monclient: shutdown 2024-03-29T11:36:38.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.098+0000 7f3575fd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:38.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.098+0000 7f3575fd7640 1 -- 172.21.15.37:0/1152442455 >> v1:172.21.15.37:6790/0 conn(0x7f35700af8e0 legacy=0x7f3570149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.102+0000 7f3575fd7640 1 -- 172.21.15.37:0/1152442455 shutdown_connections 2024-03-29T11:36:38.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.102+0000 7f3575fd7640 1 -- 172.21.15.37:0/1152442455 >> 172.21.15.37:0/1152442455 conn(0x7f357006b660 msgr2=0x7f357006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:38.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.102+0000 7f3575fd7640 1 -- 172.21.15.37:0/1152442455 shutdown_connections 2024-03-29T11:36:38.102 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.103+0000 7f3575fd7640 1 -- 172.21.15.37:0/1152442455 wait complete. 2024-03-29T11:36:38.103 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.104+0000 7f3575fd7640 10 monclient: build_initial_monmap 2024-03-29T11:36:38.103 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.104+0000 7f3575fd7640 10 monclient: monmap: 2024-03-29T11:36:38.103 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:38.104 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:38.104 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:36:38.104 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:36:38.104 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:38.104 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:38.104 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:38.104 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:36:38.104 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:36:38.104 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:38.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.104+0000 7f3575fd7640 1 librados: starting msgr at 2024-03-29T11:36:38.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.104+0000 7f3575fd7640 1 librados: starting objecter 2024-03-29T11:36:38.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.104+0000 7f3575fd7640 1 Processor -- start 2024-03-29T11:36:38.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.105+0000 7f3575fd7640 1 -- start start 2024-03-29T11:36:38.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.105+0000 7f3575fd7640 1 librados: setting wanted keys 2024-03-29T11:36:38.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.105+0000 7f3575fd7640 1 librados: calling monclient init 2024-03-29T11:36:38.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.105+0000 7f3575fd7640 10 monclient: init 2024-03-29T11:36:38.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.105+0000 7f3575fd7640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:38.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.105+0000 7f3575fd7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:38.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.105+0000 7f3575fd7640 10 monclient(hunting): picked mon.noname-c con 0x7f35700af8e0 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:38.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.105+0000 7f3575fd7640 10 monclient(hunting): picked mon.noname-b con 0x7f35700b0860 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:38.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.105+0000 7f3575fd7640 10 monclient(hunting): picked mon.noname-a con 0x7f357013ef30 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:38.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.105+0000 7f3575fd7640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35700b3d70 con 0x7f357013ef30 2024-03-29T11:36:38.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.105+0000 7f3575fd7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3570136350 con 0x7f35700b0860 2024-03-29T11:36:38.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.105+0000 7f3575fd7640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35700b2f50 con 0x7f35700af8e0 2024-03-29T11:36:38.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.105+0000 7f3575fd7640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:38.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.105+0000 7f3575fd7640 10 monclient(hunting): authenticate will time out at 977.554821s 2024-03-29T11:36:38.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.109+0000 7f356ffff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f357013ef30 0x7f35701df140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:35785/0 (socket says 172.21.15.37:35785) 2024-03-29T11:36:38.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.109+0000 7f356ffff640 1 -- 172.21.15.37:0/1596618595 learned_addr learned my addr 172.21.15.37:0/1596618595 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:38.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.111+0000 7f3557fff640 10 client.?.objecter ms_handle_connect 0x7f357013ef30 2024-03-29T11:36:38.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.111+0000 7f3557fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:38.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.111+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546155923 0 0) 0x7f35700b3d70 con 0x7f357013ef30 2024-03-29T11:36:38.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.111+0000 7f3557fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.111+0000 7f3557fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.111+0000 7f3557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.111+0000 7f3557fff640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:36:38.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.111+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f355c004eb0 con 0x7f357013ef30 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.111+0000 7f3557fff640 10 client.?.objecter ms_handle_connect 0x7f35700af8e0 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.111+0000 7f3557fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1955855681 0 0) 0x7f355c004eb0 con 0x7f357013ef30 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35700b3d70 con 0x7f357013ef30 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3568001bf0 con 0x7f357013ef30 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 10 monclient(hunting): dump: 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:38.113 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:38.114 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:38.114 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:38.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:38.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248677508 0 0) 0x7f35700b2f50 con 0x7f35700af8e0 2024-03-29T11:36:38.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:36:38.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f355c004eb0 con 0x7f35700af8e0 2024-03-29T11:36:38.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1834327672 0 0) 0x7f35700b3d70 con 0x7f357013ef30 2024-03-29T11:36:38.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 >> v1:172.21.15.37:6790/0 conn(0x7f35700af8e0 legacy=0x7f3570137d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.112+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 >> v1:172.21.15.146:6789/0 conn(0x7f35700b0860 legacy=0x7f35701dea30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:38.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3557fff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:38.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3557fff640 1 monclient: found mon.a 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3557fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3557fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35700b4290 con 0x7f357013ef30 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3557fff640 10 monclient: _finish_auth 0 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3557fff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3557fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3575fd7640 5 monclient: authenticate success, global_id 4275 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3575fd7640 10 monclient: _renew_subs 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3575fd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3575fd7640 1 -- 172.21.15.37:0/1596618595 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35700b3730 con 0x7f357013ef30 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3575fd7640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3575fd7640 10 monclient: _renew_subs 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3575fd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3575fd7640 1 -- 172.21.15.37:0/1596618595 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35701c7c20 con 0x7f357013ef30 2024-03-29T11:36:38.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3575fd7640 1 librados: init done 2024-03-29T11:36:38.116 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3568005420 con 0x7f357013ef30 2024-03-29T11:36:38.116 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3557fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:38.116 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3568005730 con 0x7f357013ef30 2024-03-29T11:36:38.116 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3557fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.116 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3557fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:38.116 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f3557fff640 10 monclient: dump: 2024-03-29T11:36:38.116 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:38.116 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:38.117 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:38.117 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:38.117 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.117 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:38.117 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:38.117 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:38.117 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:38.117 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:38.117 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.113+0000 7f356dffb640 4 set_mon_vals no callback set 2024-03-29T11:36:38.117 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.114+0000 7f3575fd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:38.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.118+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f35680015f0 con 0x7f357013ef30 2024-03-29T11:36:38.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.118+0000 7f3575fd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:38.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.118+0000 7f3575fd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:36:38.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.118+0000 7f3575fd7640 1 -- 172.21.15.37:0/1596618595 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3550005080 con 0x7f357013ef30 2024-03-29T11:36:38.119 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.119+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f35680850f0 con 0x7f357013ef30 2024-03-29T11:36:38.119 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.119+0000 7f3557fff640 10 client.4275.objecter ms_dispatch 0x7f3570000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:38.119 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.119+0000 7f3557fff640 3 client.4275.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:38.119 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.119+0000 7f3557fff640 3 client.4275.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:38.119 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.119+0000 7f3557fff640 20 client.4275.objecter dump_active .. 0 homeless 2024-03-29T11:36:38.119 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.119+0000 7f356f7fe640 10 monclient: get_auth_request con 0x7f355c07ee10 auth_method 0 2024-03-29T11:36:38.119 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.119+0000 7f3557fff640 10 client.4275.objecter ms_handle_connect 0x7f355c07ee10 2024-03-29T11:36:38.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.131+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35680509d0 con 0x7f357013ef30 2024-03-29T11:36:38.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.131+0000 7f3557fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:38.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.131+0000 7f3557fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:38.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.455+0000 7f3575fd7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:38.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.455+0000 7f3575fd7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:38.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.455+0000 7f3575fd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:36:38.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.455+0000 7f3575fd7640 1 -- 172.21.15.37:0/1596618595 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3550005330 con 0x7f357013ef30 2024-03-29T11:36:38.457 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.458+0000 7f3557fff640 1 -- 172.21.15.37:0/1596618595 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f3568052900 con 0x7f357013ef30 2024-03-29T11:36:38.457 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.458+0000 7f3557fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:38.457 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.458+0000 7f3557fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:38.458 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:36:38.458 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:38.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.461+0000 7f3575fd7640 10 librados: watch_flush enter 2024-03-29T11:36:38.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.461+0000 7f3575fd7640 10 librados: watch_flush exit 2024-03-29T11:36:38.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.461+0000 7f3575fd7640 20 client.4275.objecter shutdown clearing up homeless session... 2024-03-29T11:36:38.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.461+0000 7f3575fd7640 10 client.4275.objecter successfully canceled tick 2024-03-29T11:36:38.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.462+0000 7f3575fd7640 1 -- 172.21.15.37:0/1596618595 >> v1:172.21.15.146:6816/2899149139 conn(0x7f355c07ee10 legacy=0x7f355c0812d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.462+0000 7f3575fd7640 10 monclient: shutdown 2024-03-29T11:36:38.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.462+0000 7f3575fd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:38.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.462+0000 7f3575fd7640 1 -- 172.21.15.37:0/1596618595 >> v1:172.21.15.37:6789/0 conn(0x7f357013ef30 legacy=0x7f35701df140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.462+0000 7f3575fd7640 1 -- 172.21.15.37:0/1596618595 shutdown_connections 2024-03-29T11:36:38.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.462+0000 7f3575fd7640 1 -- 172.21.15.37:0/1596618595 >> 172.21.15.37:0/1596618595 conn(0x7f357006b660 msgr2=0x7f357013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:38.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.462+0000 7f3575fd7640 1 -- 172.21.15.37:0/1596618595 shutdown_connections 2024-03-29T11:36:38.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.462+0000 7f3575fd7640 1 -- 172.21.15.37:0/1596618595 wait complete. 2024-03-29T11:36:38.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:38.463+0000 7f3575fd7640 1 librados: shutdown 2024-03-29T11:36:38.479 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:45.286 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:45.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.439+0000 7fca6d084640 10 monclient: get_monmap_and_config 2024-03-29T11:36:45.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.439+0000 7fca6d084640 10 monclient: build_initial_monmap 2024-03-29T11:36:45.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.439+0000 7fca6d084640 10 monclient: monmap: 2024-03-29T11:36:45.439 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:45.439 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:45.439 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:36:45.440571+0000 2024-03-29T11:36:45.439 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:36:45.440571+0000 2024-03-29T11:36:45.439 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:45.439 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:45.439 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:45.439 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:36:45.440 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:36:45.440 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.440+0000 7fca6d084640 1 Processor -- start 2024-03-29T11:36:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.440+0000 7fca6d084640 1 -- start start 2024-03-29T11:36:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.440+0000 7fca6d084640 10 monclient: init 2024-03-29T11:36:45.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.441+0000 7fca6d084640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:45.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.441+0000 7fca6d084640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:45.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.441+0000 7fca6d084640 10 monclient(hunting): picked mon.noname-a con 0x7fca680b0770 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:45.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.441+0000 7fca6d084640 10 monclient(hunting): picked mon.noname-b con 0x7fca6814a700 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:45.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.441+0000 7fca6d084640 10 monclient(hunting): picked mon.noname-c con 0x7fca680ac5a0 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:45.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.441+0000 7fca6d084640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca680aba70 con 0x7fca680b0770 2024-03-29T11:36:45.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.441+0000 7fca6d084640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca680aecb0 con 0x7fca680ac5a0 2024-03-29T11:36:45.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.441+0000 7fca6d084640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca680af150 con 0x7fca6814a700 2024-03-29T11:36:45.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.441+0000 7fca6d084640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:45.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.441+0000 7fca6d084640 10 monclient(hunting): authenticate will time out at 984.890581s 2024-03-29T11:36:45.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.444+0000 7fca6759e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fca680ac5a0 0x7fca68151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:45233/0 (socket says 172.21.15.37:45233) 2024-03-29T11:36:45.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.444+0000 7fca6759e640 1 -- 172.21.15.37:0/150800420 learned_addr learned my addr 172.21.15.37:0/150800420 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:45.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.445+0000 7fca65d9b640 1 -- 172.21.15.37:0/150800420 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671964581 0 0) 0x7fca680aecb0 con 0x7fca680ac5a0 2024-03-29T11:36:45.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.445+0000 7fca65d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:36:45.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.445+0000 7fca65d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:45.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.445+0000 7fca65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:45.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.445+0000 7fca65d9b640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:36:45.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.445+0000 7fca65d9b640 1 -- 172.21.15.37:0/150800420 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca50003520 con 0x7fca680ac5a0 2024-03-29T11:36:45.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.445+0000 7fca65d9b640 1 -- 172.21.15.37:0/150800420 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1361057902 0 0) 0x7fca50003520 con 0x7fca680ac5a0 2024-03-29T11:36:45.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.446+0000 7fca65d9b640 1 -- 172.21.15.37:0/150800420 >> v1:172.21.15.37:6790/0 conn(0x7fca6814a700 legacy=0x7fca680abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:45.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca65d9b640 1 -- 172.21.15.37:0/150800420 >> v1:172.21.15.37:6789/0 conn(0x7fca680b0770 legacy=0x7fca68149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:45.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca65d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:36:45.449 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca65d9b640 1 monclient: found mon.noname-c 2024-03-29T11:36:45.449 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca65d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca65d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca65d9b640 1 -- 172.21.15.37:0/150800420 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca680ac9d0 con 0x7fca680ac5a0 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca65d9b640 10 monclient: _finish_auth 0 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca65d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca65d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca65d9b640 1 -- 172.21.15.37:0/150800420 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fca60001a30 con 0x7fca680ac5a0 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca65d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca65d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca65d9b640 10 monclient: dump: 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:45.450 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:45.451 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:45.451 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:45.451 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:45.451 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:45.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca6d084640 5 monclient: authenticate success, global_id 4291 2024-03-29T11:36:45.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.449+0000 7fca6d084640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:45.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.451+0000 7fca65d9b640 1 -- 172.21.15.37:0/150800420 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca60001c90 con 0x7fca680ac5a0 2024-03-29T11:36:45.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.451+0000 7fca65d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:45.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.451+0000 7fca65d9b640 1 -- 172.21.15.37:0/150800420 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fca600020e0 con 0x7fca680ac5a0 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.451+0000 7fca65d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.451+0000 7fca65d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.451+0000 7fca65d9b640 10 monclient: dump: 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.451+0000 7fca6d084640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:45.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.451+0000 7fca6d084640 4 set_mon_vals no callback set 2024-03-29T11:36:45.453 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.452+0000 7fca6d084640 10 monclient: shutdown 2024-03-29T11:36:45.453 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.452+0000 7fca6d084640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:45.453 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.452+0000 7fca6d084640 1 -- 172.21.15.37:0/150800420 >> v1:172.21.15.146:6789/0 conn(0x7fca680ac5a0 legacy=0x7fca68151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:45.453 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.453+0000 7fca6d084640 1 -- 172.21.15.37:0/150800420 shutdown_connections 2024-03-29T11:36:45.453 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.453+0000 7fca6d084640 1 -- 172.21.15.37:0/150800420 >> 172.21.15.37:0/150800420 conn(0x7fca6806b660 msgr2=0x7fca6806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:45.456 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.457+0000 7fca6d084640 1 -- 172.21.15.37:0/150800420 shutdown_connections 2024-03-29T11:36:45.456 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.457+0000 7fca6d084640 1 -- 172.21.15.37:0/150800420 wait complete. 2024-03-29T11:36:45.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.459+0000 7fca6d084640 10 monclient: build_initial_monmap 2024-03-29T11:36:45.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.459+0000 7fca6d084640 10 monclient: monmap: 2024-03-29T11:36:45.459 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:45.459 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:45.459 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:36:45.459 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:36:45.459 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:45.459 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:45.459 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:45.459 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:36:45.459 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:36:45.459 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:45.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.459+0000 7fca6d084640 1 librados: starting msgr at 2024-03-29T11:36:45.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.459+0000 7fca6d084640 1 librados: starting objecter 2024-03-29T11:36:45.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.459+0000 7fca6d084640 1 Processor -- start 2024-03-29T11:36:45.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.460+0000 7fca6d084640 1 -- start start 2024-03-29T11:36:45.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.460+0000 7fca6d084640 1 librados: setting wanted keys 2024-03-29T11:36:45.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.460+0000 7fca6d084640 1 librados: calling monclient init 2024-03-29T11:36:45.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.460+0000 7fca6d084640 10 monclient: init 2024-03-29T11:36:45.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.460+0000 7fca6d084640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:45.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.460+0000 7fca6d084640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:45.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.460+0000 7fca6d084640 10 monclient(hunting): picked mon.noname-c con 0x7fca680ac5a0 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:45.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.460+0000 7fca6d084640 10 monclient(hunting): picked mon.noname-b con 0x7fca680b0770 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:45.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.460+0000 7fca6d084640 10 monclient(hunting): picked mon.noname-a con 0x7fca6814a700 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:45.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.460+0000 7fca6d084640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca68120f30 con 0x7fca6814a700 2024-03-29T11:36:45.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.460+0000 7fca6d084640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca6811f2e0 con 0x7fca680b0770 2024-03-29T11:36:45.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.460+0000 7fca6d084640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca6811c190 con 0x7fca680ac5a0 2024-03-29T11:36:45.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.460+0000 7fca6d084640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:45.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.460+0000 7fca6d084640 10 monclient(hunting): authenticate will time out at 984.909879s 2024-03-29T11:36:45.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.464+0000 7fca66d9d640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fca680ac5a0 0x7fca6811f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:46409/0 (socket says 172.21.15.37:46409) 2024-03-29T11:36:45.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.464+0000 7fca66d9d640 1 -- 172.21.15.37:0/2401696642 learned_addr learned my addr 172.21.15.37:0/2401696642 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.464+0000 7fca4f7fe640 10 client.?.objecter ms_handle_connect 0x7fca680ac5a0 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.464+0000 7fca4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.464+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592296924 0 0) 0x7fca6811c190 con 0x7fca680ac5a0 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.464+0000 7fca4f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.464+0000 7fca4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.464+0000 7fca4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.464+0000 7fca4f7fe640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.464+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca54004eb0 con 0x7fca680ac5a0 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.465+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3511510948 0 0) 0x7fca54004eb0 con 0x7fca680ac5a0 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.465+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca6811c190 con 0x7fca680ac5a0 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.465+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fca58001960 con 0x7fca680ac5a0 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.465+0000 7fca4f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.465+0000 7fca4f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.465+0000 7fca4f7fe640 10 monclient(hunting): dump: 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:45.468 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:45.469 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:45.469 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:45.469 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:45.469 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:45.469 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:45.469 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:45.469 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:45.469 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:45.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.465+0000 7fca4f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:45.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.465+0000 7fca4f7fe640 10 client.?.objecter ms_handle_connect 0x7fca6814a700 2024-03-29T11:36:45.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.465+0000 7fca4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:45.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.465+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3675040516 0 0) 0x7fca6811c190 con 0x7fca680ac5a0 2024-03-29T11:36:45.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.466+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 >> v1:172.21.15.146:6789/0 conn(0x7fca680b0770 legacy=0x7fca68149000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:45.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 >> v1:172.21.15.37:6789/0 conn(0x7fca6814a700 legacy=0x7fca681e4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:45.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca4f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:45.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca4f7fe640 1 monclient: found mon.c 2024-03-29T11:36:45.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:45.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca4f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:45.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6811c320 con 0x7fca680ac5a0 2024-03-29T11:36:45.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca4f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:45.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca4f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca6d084640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca6d084640 10 monclient: _renew_subs 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca6d084640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca6d084640 1 -- 172.21.15.37:0/2401696642 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca6811ff10 con 0x7fca680ac5a0 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca6d084640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca6d084640 10 monclient: _renew_subs 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca6d084640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca6d084640 1 -- 172.21.15.37:0/2401696642 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca681498f0 con 0x7fca680ac5a0 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.469+0000 7fca6d084640 1 librados: init done 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.472+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca58001c80 con 0x7fca680ac5a0 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.472+0000 7fca4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.472+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fca58003a40 con 0x7fca680ac5a0 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.472+0000 7fca4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:45.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.472+0000 7fca64d99640 4 set_mon_vals no callback set 2024-03-29T11:36:45.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.472+0000 7fca4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:45.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.472+0000 7fca4f7fe640 10 monclient: dump: 2024-03-29T11:36:45.474 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:45.474 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:45.474 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:45.474 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:45.474 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:45.474 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:45.474 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:45.474 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:45.474 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:45.474 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:45.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.472+0000 7fca6d084640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:45.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.473+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fca58003c60 con 0x7fca680ac5a0 2024-03-29T11:36:45.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.474+0000 7fca6d084640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:45.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.474+0000 7fca6d084640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:45.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.474+0000 7fca6d084640 1 -- 172.21.15.37:0/2401696642 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca48005080 con 0x7fca680ac5a0 2024-03-29T11:36:45.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.474+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fca58084b20 con 0x7fca680ac5a0 2024-03-29T11:36:45.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.474+0000 7fca6659c640 10 monclient: get_auth_request con 0x7fca5406dbb0 auth_method 0 2024-03-29T11:36:45.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.474+0000 7fca4f7fe640 10 client.4298.objecter ms_dispatch 0x7fca68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:45.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.477+0000 7fca4f7fe640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:45.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.478+0000 7fca4f7fe640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:45.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.478+0000 7fca4f7fe640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T11:36:45.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.478+0000 7fca4f7fe640 10 client.4298.objecter ms_handle_connect 0x7fca5406dbb0 2024-03-29T11:36:45.485 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.486+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca58050400 con 0x7fca680ac5a0 2024-03-29T11:36:45.485 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.486+0000 7fca4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:45.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.486+0000 7fca4f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:45.817 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.818+0000 7fca6d084640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:45.817 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.818+0000 7fca6d084640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:45.817 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.818+0000 7fca6d084640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:45.817 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.818+0000 7fca6d084640 1 -- 172.21.15.37:0/2401696642 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca48005330 con 0x7fca680ac5a0 2024-03-29T11:36:45.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.819+0000 7fca4f7fe640 1 -- 172.21.15.37:0/2401696642 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fca58052330 con 0x7fca680ac5a0 2024-03-29T11:36:45.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.819+0000 7fca4f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:45.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.819+0000 7fca4f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:45.822 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:36:45.823 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:45.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.827+0000 7fca6d084640 10 librados: watch_flush enter 2024-03-29T11:36:45.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.827+0000 7fca6d084640 10 librados: watch_flush exit 2024-03-29T11:36:45.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.827+0000 7fca6d084640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T11:36:45.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.827+0000 7fca6d084640 10 client.4298.objecter successfully canceled tick 2024-03-29T11:36:45.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.828+0000 7fca6d084640 1 -- 172.21.15.37:0/2401696642 >> v1:172.21.15.146:6816/2899149139 conn(0x7fca5406dbb0 legacy=0x7fca54070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:45.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.828+0000 7fca6d084640 10 monclient: shutdown 2024-03-29T11:36:45.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.828+0000 7fca6d084640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:45.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.828+0000 7fca6d084640 1 -- 172.21.15.37:0/2401696642 >> v1:172.21.15.37:6790/0 conn(0x7fca680ac5a0 legacy=0x7fca6811f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:45.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.828+0000 7fca6d084640 1 -- 172.21.15.37:0/2401696642 shutdown_connections 2024-03-29T11:36:45.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.828+0000 7fca6d084640 1 -- 172.21.15.37:0/2401696642 >> 172.21.15.37:0/2401696642 conn(0x7fca6806b660 msgr2=0x7fca680b2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:45.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.828+0000 7fca6d084640 1 -- 172.21.15.37:0/2401696642 shutdown_connections 2024-03-29T11:36:45.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.828+0000 7fca6d084640 1 -- 172.21.15.37:0/2401696642 wait complete. 2024-03-29T11:36:45.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:45.828+0000 7fca6d084640 1 librados: shutdown 2024-03-29T11:36:45.845 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:52.652 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:52.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.805+0000 7fa8aa4ba640 10 monclient: get_monmap_and_config 2024-03-29T11:36:52.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.805+0000 7fa8aa4ba640 10 monclient: build_initial_monmap 2024-03-29T11:36:52.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.805+0000 7fa8aa4ba640 10 monclient: monmap: 2024-03-29T11:36:52.810 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:52.810 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:52.810 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:36:52.806086+0000 2024-03-29T11:36:52.810 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:36:52.806086+0000 2024-03-29T11:36:52.810 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:52.811 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:52.811 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:52.811 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:36:52.811 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:36:52.811 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:52.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.806+0000 7fa8aa4ba640 1 Processor -- start 2024-03-29T11:36:52.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.806+0000 7fa8aa4ba640 1 -- start start 2024-03-29T11:36:52.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.806+0000 7fa8aa4ba640 10 monclient: init 2024-03-29T11:36:52.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.807+0000 7fa8aa4ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:52.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.807+0000 7fa8aa4ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:52.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.807+0000 7fa8aa4ba640 10 monclient(hunting): picked mon.noname-c con 0x7fa8a40ad6c0 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:52.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.807+0000 7fa8aa4ba640 10 monclient(hunting): picked mon.noname-a con 0x7fa8a4153210 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:52.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.807+0000 7fa8aa4ba640 10 monclient(hunting): picked mon.noname-b con 0x7fa8a40ab380 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:52.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.807+0000 7fa8aa4ba640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a40a94e0 con 0x7fa8a4153210 2024-03-29T11:36:52.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.807+0000 7fa8aa4ba640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a40af3a0 con 0x7fa8a40ad6c0 2024-03-29T11:36:52.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.807+0000 7fa8aa4ba640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a40afb60 con 0x7fa8a40ab380 2024-03-29T11:36:52.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.807+0000 7fa8aa4ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:52.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.807+0000 7fa8aa4ba640 10 monclient(hunting): authenticate will time out at 992.256540s 2024-03-29T11:36:52.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.810+0000 7fa8a3fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa8a40ad6c0 0x7fa8a414fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:36713/0 (socket says 172.21.15.37:36713) 2024-03-29T11:36:52.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.810+0000 7fa8a3fff640 1 -- 172.21.15.37:0/844014188 learned_addr learned my addr 172.21.15.37:0/844014188 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:52.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.812+0000 7fa8a2ffd640 1 -- 172.21.15.37:0/844014188 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893218755 0 0) 0x7fa8a40af3a0 con 0x7fa8a40ad6c0 2024-03-29T11:36:52.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.812+0000 7fa8a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:52.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.812+0000 7fa8a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:52.818 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.812+0000 7fa8a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:52.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.812+0000 7fa8a2ffd640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:36:52.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.812+0000 7fa8a2ffd640 1 -- 172.21.15.37:0/844014188 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa890003520 con 0x7fa8a40ad6c0 2024-03-29T11:36:52.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 1 -- 172.21.15.37:0/844014188 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092079626 0 0) 0x7fa8a40a94e0 con 0x7fa8a4153210 2024-03-29T11:36:52.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:52.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:52.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:52.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:36:52.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 1 -- 172.21.15.37:0/844014188 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a40af3a0 con 0x7fa8a4153210 2024-03-29T11:36:52.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 1 -- 172.21.15.37:0/844014188 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 238819130 0 0) 0x7fa8a40af3a0 con 0x7fa8a4153210 2024-03-29T11:36:52.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 1 -- 172.21.15.37:0/844014188 >> v1:172.21.15.37:6790/0 conn(0x7fa8a40ab380 legacy=0x7fa8a4152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:52.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 1 -- 172.21.15.37:0/844014188 >> v1:172.21.15.146:6789/0 conn(0x7fa8a40ad6c0 legacy=0x7fa8a414fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:52.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:52.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 1 monclient: found mon.noname-a 2024-03-29T11:36:52.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:52.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:36:52.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 1 -- 172.21.15.37:0/844014188 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a40aab80 con 0x7fa8a4153210 2024-03-29T11:36:52.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:52.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:52.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:52.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 1 -- 172.21.15.37:0/844014188 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa894001670 con 0x7fa8a4153210 2024-03-29T11:36:52.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:52.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:52.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 10 monclient: dump: 2024-03-29T11:36:52.823 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:52.824 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:52.824 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:52.824 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:52.824 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:52.824 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:52.825 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:52.825 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:52.825 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:52.825 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:52.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8aa4ba640 5 monclient: authenticate success, global_id 4281 2024-03-29T11:36:52.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8aa4ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:52.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 1 -- 172.21.15.37:0/844014188 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8940017d0 con 0x7fa8a4153210 2024-03-29T11:36:52.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:52.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8aa4ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:52.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.813+0000 7fa8aa4ba640 4 set_mon_vals no callback set 2024-03-29T11:36:52.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.814+0000 7fa8a2ffd640 1 -- 172.21.15.37:0/844014188 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa894001c20 con 0x7fa8a4153210 2024-03-29T11:36:52.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.814+0000 7fa8a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:52.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.814+0000 7fa8a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:52.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.814+0000 7fa8a2ffd640 10 monclient: dump: 2024-03-29T11:36:52.827 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:52.828 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:52.828 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:52.828 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:52.828 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:52.828 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:52.829 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:52.829 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:52.829 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:52.829 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:52.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.814+0000 7fa8aa4ba640 10 monclient: shutdown 2024-03-29T11:36:52.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.814+0000 7fa8aa4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:52.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.814+0000 7fa8aa4ba640 1 -- 172.21.15.37:0/844014188 >> v1:172.21.15.37:6789/0 conn(0x7fa8a4153210 legacy=0x7fa8a41578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:52.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.815+0000 7fa8aa4ba640 1 -- 172.21.15.37:0/844014188 shutdown_connections 2024-03-29T11:36:52.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.815+0000 7fa8aa4ba640 1 -- 172.21.15.37:0/844014188 >> 172.21.15.37:0/844014188 conn(0x7fa8a40b35e0 msgr2=0x7fa8a40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:52.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.816+0000 7fa8aa4ba640 1 -- 172.21.15.37:0/844014188 shutdown_connections 2024-03-29T11:36:52.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.816+0000 7fa8aa4ba640 1 -- 172.21.15.37:0/844014188 wait complete. 2024-03-29T11:36:52.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.817+0000 7fa8aa4ba640 10 monclient: build_initial_monmap 2024-03-29T11:36:52.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.817+0000 7fa8aa4ba640 10 monclient: monmap: 2024-03-29T11:36:52.831 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:36:52.832 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:52.832 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:36:52.832 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:36:52.832 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:52.832 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:52.833 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:36:52.833 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:36:52.833 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:36:52.833 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:52.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.817+0000 7fa8aa4ba640 1 librados: starting msgr at 2024-03-29T11:36:52.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.817+0000 7fa8aa4ba640 1 librados: starting objecter 2024-03-29T11:36:52.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.817+0000 7fa8aa4ba640 1 Processor -- start 2024-03-29T11:36:52.842 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.818+0000 7fa8aa4ba640 1 -- start start 2024-03-29T11:36:52.842 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.818+0000 7fa8aa4ba640 1 librados: setting wanted keys 2024-03-29T11:36:52.842 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.818+0000 7fa8aa4ba640 1 librados: calling monclient init 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.818+0000 7fa8aa4ba640 10 monclient: init 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.819+0000 7fa8aa4ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.819+0000 7fa8aa4ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.819+0000 7fa8aa4ba640 10 monclient(hunting): picked mon.noname-b con 0x7fa8a40ab380 addr v1:172.21.15.146:6789/0 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.819+0000 7fa8aa4ba640 10 monclient(hunting): picked mon.noname-a con 0x7fa8a40ad6c0 addr v1:172.21.15.37:6789/0 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.819+0000 7fa8aa4ba640 10 monclient(hunting): picked mon.noname-c con 0x7fa8a4153210 addr v1:172.21.15.37:6790/0 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.819+0000 7fa8aa4ba640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a406f440 con 0x7fa8a40ad6c0 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.819+0000 7fa8aa4ba640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a406f720 con 0x7fa8a40ab380 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.819+0000 7fa8aa4ba640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a40723c0 con 0x7fa8a4153210 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.819+0000 7fa8aa4ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.819+0000 7fa8aa4ba640 10 monclient(hunting): authenticate will time out at 992.268765s 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.824+0000 7fa8a37fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa8a40ad6c0 0x7fa8a41f0360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49155/0 (socket says 172.21.15.37:49155) 2024-03-29T11:36:52.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.824+0000 7fa8a37fe640 1 -- 172.21.15.37:0/1991334571 learned_addr learned my addr 172.21.15.37:0/1991334571 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:36:52.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.827+0000 7fa8a0ff9640 10 client.?.objecter ms_handle_connect 0x7fa8a40ad6c0 2024-03-29T11:36:52.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.827+0000 7fa8a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:52.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 client.?.objecter ms_handle_connect 0x7fa8a4153210 2024-03-29T11:36:52.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:52.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 client.?.objecter ms_handle_connect 0x7fa8a40ab380 2024-03-29T11:36:52.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:52.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762297682 0 0) 0x7fa8a40723c0 con 0x7fa8a4153210 2024-03-29T11:36:52.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:52.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:52.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:52.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:36:52.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa884004eb0 con 0x7fa8a4153210 2024-03-29T11:36:52.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534845748 0 0) 0x7fa8a406f720 con 0x7fa8a40ab380 2024-03-29T11:36:52.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:52.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:52.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:52.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:36:52.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a40723c0 con 0x7fa8a40ab380 2024-03-29T11:36:52.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611152401 0 0) 0x7fa8a406f440 con 0x7fa8a40ad6c0 2024-03-29T11:36:52.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:52.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:52.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:52.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:36:52.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a406f720 con 0x7fa8a40ad6c0 2024-03-29T11:36:52.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3096288593 0 0) 0x7fa884004eb0 con 0x7fa8a4153210 2024-03-29T11:36:52.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8a406f440 con 0x7fa8a4153210 2024-03-29T11:36:52.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa88c0017f0 con 0x7fa8a4153210 2024-03-29T11:36:52.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:52.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:52.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:52.850 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:52.850 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:52.850 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:52.850 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:52.850 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:52.850 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:52.851 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:52.851 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:52.851 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:52.851 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:52.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.830+0000 7fa8a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:52.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2439364298 0 0) 0x7fa8a40723c0 con 0x7fa8a40ab380 2024-03-29T11:36:52.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa884004eb0 con 0x7fa8a40ab380 2024-03-29T11:36:52.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa89c001c50 con 0x7fa8a40ab380 2024-03-29T11:36:52.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:52.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:52.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:52.852 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2414428761 0 0) 0x7fa8a406f440 con 0x7fa8a4153210 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 >> v1:172.21.15.146:6789/0 conn(0x7fa8a40ab380 legacy=0x7fa8a406fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 >> v1:172.21.15.37:6789/0 conn(0x7fa8a40ad6c0 legacy=0x7fa8a41f0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 1 monclient: found mon.c 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a4072550 con 0x7fa8a4153210 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8aa4ba640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8aa4ba640 10 monclient: _renew_subs 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8aa4ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8aa4ba640 1 -- 172.21.15.37:0/1991334571 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a4070530 con 0x7fa8a4153210 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8aa4ba640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8aa4ba640 10 monclient: _renew_subs 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8aa4ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8aa4ba640 1 -- 172.21.15.37:0/1991334571 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a406e980 con 0x7fa8a4153210 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.831+0000 7fa8aa4ba640 1 librados: init done 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.832+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa88c001d00 con 0x7fa8a4153210 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.832+0000 7fa8a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.832+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa88c003790 con 0x7fa8a4153210 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.832+0000 7fa8a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.832+0000 7fa8a0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.832+0000 7fa8a0ff9640 10 monclient: dump: 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.832+0000 7fa8a1ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.835+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fa88c0038f0 con 0x7fa8a4153210 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.835+0000 7fa89a7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.835+0000 7fa89a7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.835+0000 7fa89a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.835+0000 7fa89a7fc640 1 -- 172.21.15.37:0/1991334571 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa888005080 con 0x7fa8a4153210 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.836+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fa88c083750 con 0x7fa8a4153210 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.836+0000 7fa8a0ff9640 10 client.4301.objecter ms_dispatch 0x7fa8a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.836+0000 7fa8a0ff9640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.836+0000 7fa8a0ff9640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.836+0000 7fa8a0ff9640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.836+0000 7fa8a3fff640 10 monclient: get_auth_request con 0x7fa88406df90 auth_method 0 2024-03-29T11:36:52.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.836+0000 7fa8a0ff9640 10 client.4301.objecter ms_handle_connect 0x7fa88406df90 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.845+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa88c04f030 con 0x7fa8a4153210 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.845+0000 7fa8a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:52.845+0000 7fa8a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:53.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.192+0000 7fa89a7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:53.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.192+0000 7fa89a7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:53.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.192+0000 7fa89a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:36:53.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.192+0000 7fa89a7fc640 1 -- 172.21.15.37:0/1991334571 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa888005330 con 0x7fa8a4153210 2024-03-29T11:36:53.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.193+0000 7fa8a0ff9640 1 -- 172.21.15.37:0/1991334571 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fa88c050f60 con 0x7fa8a4153210 2024-03-29T11:36:53.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.193+0000 7fa8a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:53.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.193+0000 7fa8a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:53.194 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:36:53.194 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:53.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.198+0000 7fa89a7fc640 10 librados: watch_flush enter 2024-03-29T11:36:53.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.198+0000 7fa89a7fc640 10 librados: watch_flush exit 2024-03-29T11:36:53.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.198+0000 7fa89a7fc640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-29T11:36:53.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.198+0000 7fa89a7fc640 10 client.4301.objecter successfully canceled tick 2024-03-29T11:36:53.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.198+0000 7fa89a7fc640 1 -- 172.21.15.37:0/1991334571 >> v1:172.21.15.146:6816/2899149139 conn(0x7fa88406df90 legacy=0x7fa884070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:53.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.198+0000 7fa89a7fc640 10 monclient: shutdown 2024-03-29T11:36:53.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.198+0000 7fa89a7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:53.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.198+0000 7fa89a7fc640 1 -- 172.21.15.37:0/1991334571 >> v1:172.21.15.37:6790/0 conn(0x7fa8a4153210 legacy=0x7fa8a41f81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:53.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.198+0000 7fa89a7fc640 1 -- 172.21.15.37:0/1991334571 shutdown_connections 2024-03-29T11:36:53.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.198+0000 7fa89a7fc640 1 -- 172.21.15.37:0/1991334571 >> 172.21.15.37:0/1991334571 conn(0x7fa8a40b35e0 msgr2=0x7fa8a40a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:53.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.199+0000 7fa89a7fc640 1 -- 172.21.15.37:0/1991334571 shutdown_connections 2024-03-29T11:36:53.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.199+0000 7fa89a7fc640 1 -- 172.21.15.37:0/1991334571 wait complete. 2024-03-29T11:36:53.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:36:53.199+0000 7fa89a7fc640 1 librados: shutdown 2024-03-29T11:36:53.220 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:00.027 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:00.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.178+0000 7f8e43fff640 10 monclient: get_monmap_and_config 2024-03-29T11:37:00.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.178+0000 7f8e43fff640 10 monclient: build_initial_monmap 2024-03-29T11:37:00.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.178+0000 7f8e43fff640 10 monclient: monmap: 2024-03-29T11:37:00.178 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:00.178 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:00.178 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:37:00.179713+0000 2024-03-29T11:37:00.178 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:37:00.179713+0000 2024-03-29T11:37:00.178 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:00.179 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:00.179 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:00.179 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:37:00.179 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:37:00.179 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:00.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.179+0000 7f8e43fff640 1 Processor -- start 2024-03-29T11:37:00.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.180+0000 7f8e43fff640 1 -- start start 2024-03-29T11:37:00.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.180+0000 7f8e43fff640 10 monclient: init 2024-03-29T11:37:00.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.181+0000 7f8e43fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:00.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.181+0000 7f8e43fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:00.183 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.184+0000 7f8e42ffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8e3c037230 0x7f8e3c036080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:37481/0 (socket says 172.21.15.37:37481) 2024-03-29T11:37:00.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.186+0000 7f8e42ffd640 1 -- 172.21.15.37:0/1687038181 learned_addr learned my addr 172.21.15.37:0/1687038181 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:00.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.186+0000 7f8e43fff640 10 monclient(hunting): picked mon.noname-a con 0x7f8e3c037230 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:00.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.186+0000 7f8e43fff640 10 monclient(hunting): picked mon.noname-b con 0x7f8e3c033a80 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:00.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.186+0000 7f8e43fff640 10 monclient(hunting): picked mon.noname-c con 0x7f8e3c0d4770 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:00.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.186+0000 7f8e43fff640 1 -- 172.21.15.37:0/1687038181 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3c035bf0 con 0x7f8e3c037230 2024-03-29T11:37:00.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.186+0000 7f8e43fff640 1 -- 172.21.15.37:0/1687038181 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3c02c150 con 0x7f8e3c0d4770 2024-03-29T11:37:00.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.186+0000 7f8e43fff640 1 -- 172.21.15.37:0/1687038181 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3c0356f0 con 0x7f8e3c033a80 2024-03-29T11:37:00.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.186+0000 7f8e43fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:00.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.186+0000 7f8e43fff640 10 monclient(hunting): authenticate will time out at 999.635306s 2024-03-29T11:37:00.187 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.188+0000 7f8e41ffb640 1 -- 172.21.15.37:0/1687038181 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843285530 0 0) 0x7f8e3c035bf0 con 0x7f8e3c037230 2024-03-29T11:37:00.187 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.188+0000 7f8e41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:00.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.188+0000 7f8e41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:00.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.188+0000 7f8e41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:00.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.188+0000 7f8e41ffb640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:37:00.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.188+0000 7f8e41ffb640 1 -- 172.21.15.37:0/1687038181 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e2c003520 con 0x7f8e3c037230 2024-03-29T11:37:00.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 1 -- 172.21.15.37:0/1687038181 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4092743576 0 0) 0x7f8e2c003520 con 0x7f8e3c037230 2024-03-29T11:37:00.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 1 -- 172.21.15.37:0/1687038181 >> v1:172.21.15.37:6790/0 conn(0x7f8e3c033a80 legacy=0x7f8e3c0cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:00.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 1 -- 172.21.15.37:0/1687038181 >> v1:172.21.15.146:6789/0 conn(0x7f8e3c0d4770 legacy=0x7f8e3c0d6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:00.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:00.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 1 monclient: found mon.noname-a 2024-03-29T11:37:00.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:00.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:37:00.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 1 -- 172.21.15.37:0/1687038181 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3c035880 con 0x7f8e3c037230 2024-03-29T11:37:00.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:00.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:00.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:00.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 1 -- 172.21.15.37:0/1687038181 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8e4404f730 con 0x7f8e3c037230 2024-03-29T11:37:00.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:00.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.191+0000 7f8e41ffb640 10 monclient: dump: 2024-03-29T11:37:00.192 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:00.192 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:00.192 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:00.192 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:00.193 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.193 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:00.193 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:00.193 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:00.193 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:00.193 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:00.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.192+0000 7f8e43fff640 5 monclient: authenticate success, global_id 4287 2024-03-29T11:37:00.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.192+0000 7f8e43fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:00.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.193+0000 7f8e41ffb640 1 -- 172.21.15.37:0/1687038181 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e44075f10 con 0x7f8e3c037230 2024-03-29T11:37:00.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.193+0000 7f8e41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:00.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.194+0000 7f8e41ffb640 1 -- 172.21.15.37:0/1687038181 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8e440758b0 con 0x7f8e3c037230 2024-03-29T11:37:00.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.194+0000 7f8e41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.194+0000 7f8e41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:00.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.194+0000 7f8e41ffb640 10 monclient: dump: 2024-03-29T11:37:00.196 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:00.196 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:00.196 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:00.196 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:00.196 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.196 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:00.196 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:00.196 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:00.196 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:00.197 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:00.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.194+0000 7f8e43fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:00.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.194+0000 7f8e43fff640 4 set_mon_vals no callback set 2024-03-29T11:37:00.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.195+0000 7f8e43fff640 10 monclient: shutdown 2024-03-29T11:37:00.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.195+0000 7f8e43fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:00.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.195+0000 7f8e43fff640 1 -- 172.21.15.37:0/1687038181 >> v1:172.21.15.37:6789/0 conn(0x7f8e3c037230 legacy=0x7f8e3c036080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.195+0000 7f8e43fff640 1 -- 172.21.15.37:0/1687038181 shutdown_connections 2024-03-29T11:37:00.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.195+0000 7f8e43fff640 1 -- 172.21.15.37:0/1687038181 >> 172.21.15.37:0/1687038181 conn(0x7f8e3c03a590 msgr2=0x7f8e3c0367e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:00.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.195+0000 7f8e43fff640 1 -- 172.21.15.37:0/1687038181 shutdown_connections 2024-03-29T11:37:00.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.196+0000 7f8e43fff640 1 -- 172.21.15.37:0/1687038181 wait complete. 2024-03-29T11:37:00.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.197+0000 7f8e43fff640 10 monclient: build_initial_monmap 2024-03-29T11:37:00.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.197+0000 7f8e43fff640 10 monclient: monmap: 2024-03-29T11:37:00.199 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.197+0000 7f8e43fff640 1 librados: starting msgr at 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.197+0000 7f8e43fff640 1 librados: starting objecter 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.198+0000 7f8e43fff640 1 Processor -- start 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.198+0000 7f8e43fff640 1 -- start start 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.198+0000 7f8e43fff640 1 librados: setting wanted keys 2024-03-29T11:37:00.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.198+0000 7f8e43fff640 1 librados: calling monclient init 2024-03-29T11:37:00.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.198+0000 7f8e43fff640 10 monclient: init 2024-03-29T11:37:00.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.199+0000 7f8e43fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:00.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.199+0000 7f8e43fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:00.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.200+0000 7f8e43fff640 10 monclient(hunting): picked mon.noname-a con 0x7f8e3c033a80 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:00.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.200+0000 7f8e43fff640 10 monclient(hunting): picked mon.noname-c con 0x7f8e3c037230 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:00.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.200+0000 7f8e43fff640 10 monclient(hunting): picked mon.noname-b con 0x7f8e3c0d4770 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:00.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.200+0000 7f8e43fff640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3c169250 con 0x7f8e3c033a80 2024-03-29T11:37:00.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.200+0000 7f8e43fff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3c0d4470 con 0x7f8e3c0d4770 2024-03-29T11:37:00.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.200+0000 7f8e43fff640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3c169e80 con 0x7f8e3c037230 2024-03-29T11:37:00.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.200+0000 7f8e43fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:00.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.200+0000 7f8e43fff640 10 monclient(hunting): authenticate will time out at 999.649054s 2024-03-29T11:37:00.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.206+0000 7f8e42ffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8e3c033a80 0x7f8e3c169770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:37675/0 (socket says 172.21.15.37:37675) 2024-03-29T11:37:00.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.206+0000 7f8e42ffd640 1 -- 172.21.15.37:0/1570569194 learned_addr learned my addr 172.21.15.37:0/1570569194 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:00.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.206+0000 7f8e437fe640 1 --1- 172.21.15.37:0/1570569194 >> v1:172.21.15.146:6789/0 conn(0x7f8e3c0d4770 0x7f8e3c1749d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:45001/0 (socket says 172.21.15.37:45001) 2024-03-29T11:37:00.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 10 client.?.objecter ms_handle_connect 0x7f8e3c0d4770 2024-03-29T11:37:00.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 10 client.?.objecter ms_handle_connect 0x7f8e3c033a80 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 10 client.?.objecter ms_handle_connect 0x7f8e3c037230 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918888382 0 0) 0x7f8e3c169250 con 0x7f8e3c033a80 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e30004eb0 con 0x7f8e3c033a80 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500923583 0 0) 0x7f8e3c169e80 con 0x7f8e3c037230 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:00.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:37:00.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.207+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e3c169250 con 0x7f8e3c037230 2024-03-29T11:37:00.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 275769594 0 0) 0x7f8e30004eb0 con 0x7f8e3c033a80 2024-03-29T11:37:00.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e3c169e80 con 0x7f8e3c033a80 2024-03-29T11:37:00.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1402042067 0 0) 0x7f8e3c169250 con 0x7f8e3c037230 2024-03-29T11:37:00.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e30004eb0 con 0x7f8e3c037230 2024-03-29T11:37:00.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8e4404f400 con 0x7f8e3c033a80 2024-03-29T11:37:00.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:00.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 10 monclient(hunting): dump: 2024-03-29T11:37:00.209 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:00.209 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:00.209 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:00.209 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:00.210 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.210 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:00.210 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:00.210 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:00.210 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:00.210 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:00.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:00.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8e38001720 con 0x7f8e3c037230 2024-03-29T11:37:00.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:00.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 10 monclient(hunting): dump: 2024-03-29T11:37:00.210 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:00.210 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:00.210 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:00.211 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:00.211 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.211 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:00.211 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:00.211 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:00.211 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:00.211 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:00.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4126077155 0 0) 0x7f8e3c169e80 con 0x7f8e3c033a80 2024-03-29T11:37:00.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 >> v1:172.21.15.37:6790/0 conn(0x7f8e3c037230 legacy=0x7f8e3c0c56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 >> v1:172.21.15.146:6789/0 conn(0x7f8e3c0d4770 legacy=0x7f8e3c1749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:00.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 1 monclient: found mon.a 2024-03-29T11:37:00.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:00.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:37:00.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3c16a010 con 0x7f8e3c033a80 2024-03-29T11:37:00.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:00.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e43fff640 5 monclient: authenticate success, global_id 4290 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e43fff640 10 monclient: _renew_subs 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e43fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e43fff640 1 -- 172.21.15.37:0/1570569194 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e3c0d3fb0 con 0x7f8e3c033a80 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e43fff640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e43fff640 10 monclient: _renew_subs 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e43fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e43fff640 1 -- 172.21.15.37:0/1570569194 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e3c0d18f0 con 0x7f8e3c033a80 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.208+0000 7f8e43fff640 1 librados: init done 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.210+0000 7f8e43fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.210+0000 7f8e43fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.210+0000 7f8e43fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.210+0000 7f8e43fff640 1 -- 172.21.15.37:0/1570569194 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e3c169490 con 0x7f8e3c033a80 2024-03-29T11:37:00.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.211+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e4404fdd0 con 0x7f8e3c033a80 2024-03-29T11:37:00.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.211+0000 7f8e2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:00.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.211+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8e44067bf0 con 0x7f8e3c033a80 2024-03-29T11:37:00.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.211+0000 7f8e2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.211+0000 7f8e2b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:00.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.211+0000 7f8e2b7fe640 10 monclient: dump: 2024-03-29T11:37:00.214 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:00.214 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:00.214 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:00.214 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:00.214 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.214 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:00.214 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:00.214 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:00.214 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:00.215 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:00.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.211+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f8e44067e10 con 0x7f8e3c033a80 2024-03-29T11:37:00.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.212+0000 7f8e417fa640 4 set_mon_vals no callback set 2024-03-29T11:37:00.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.212+0000 7f8e427fc640 10 monclient: get_auth_request con 0x7f8e3006dfa0 auth_method 0 2024-03-29T11:37:00.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.214+0000 7f8e2b7fe640 10 client.4290.objecter ms_handle_connect 0x7f8e3006dfa0 2024-03-29T11:37:00.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.214+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f8e440cf3c0 con 0x7f8e3c033a80 2024-03-29T11:37:00.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.214+0000 7f8e2b7fe640 10 client.4290.objecter ms_dispatch 0x7f8e44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:00.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.214+0000 7f8e2b7fe640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:00.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.214+0000 7f8e2b7fe640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:00.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.214+0000 7f8e2b7fe640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-29T11:37:00.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.223+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e3c169490 con 0x7f8e3c033a80 2024-03-29T11:37:00.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.223+0000 7f8e2b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:00.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.223+0000 7f8e2b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:00.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.547+0000 7f8e43fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:00.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.547+0000 7f8e43fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:00.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.547+0000 7f8e43fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:37:00.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.547+0000 7f8e43fff640 1 -- 172.21.15.37:0/1570569194 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e3c1693e0 con 0x7f8e3c033a80 2024-03-29T11:37:00.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.548+0000 7f8e2b7fe640 1 -- 172.21.15.37:0/1570569194 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f8e440caed0 con 0x7f8e3c033a80 2024-03-29T11:37:00.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.548+0000 7f8e2b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:00.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.548+0000 7f8e2b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:00.549 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:37:00.549 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:00.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.553+0000 7f8e43fff640 10 librados: watch_flush enter 2024-03-29T11:37:00.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.553+0000 7f8e43fff640 10 librados: watch_flush exit 2024-03-29T11:37:00.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.553+0000 7f8e43fff640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-29T11:37:00.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.553+0000 7f8e43fff640 10 client.4290.objecter successfully canceled tick 2024-03-29T11:37:00.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.553+0000 7f8e43fff640 1 -- 172.21.15.37:0/1570569194 >> v1:172.21.15.146:6816/2899149139 conn(0x7f8e3006dfa0 legacy=0x7f8e30070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.553+0000 7f8e43fff640 10 monclient: shutdown 2024-03-29T11:37:00.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.553+0000 7f8e43fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:00.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.553+0000 7f8e43fff640 1 -- 172.21.15.37:0/1570569194 >> v1:172.21.15.37:6789/0 conn(0x7f8e3c033a80 legacy=0x7f8e3c169770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.557+0000 7f8e43fff640 1 -- 172.21.15.37:0/1570569194 shutdown_connections 2024-03-29T11:37:00.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.557+0000 7f8e43fff640 1 -- 172.21.15.37:0/1570569194 >> 172.21.15.37:0/1570569194 conn(0x7f8e3c03a590 msgr2=0x7f8e3c004890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:00.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.557+0000 7f8e43fff640 1 -- 172.21.15.37:0/1570569194 shutdown_connections 2024-03-29T11:37:00.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.557+0000 7f8e43fff640 1 -- 172.21.15.37:0/1570569194 wait complete. 2024-03-29T11:37:00.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:00.558+0000 7f8e43fff640 1 librados: shutdown 2024-03-29T11:37:00.574 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:07.381 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:07.533 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.533+0000 7f72a9920640 10 monclient: get_monmap_and_config 2024-03-29T11:37:07.533 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.533+0000 7f72a9920640 10 monclient: build_initial_monmap 2024-03-29T11:37:07.533 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.533+0000 7f72a9920640 10 monclient: monmap: 2024-03-29T11:37:07.533 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:07.533 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:07.533 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:37:07.534567+0000 2024-03-29T11:37:07.533 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:37:07.534567+0000 2024-03-29T11:37:07.534 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:07.534 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:07.534 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:07.534 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:37:07.534 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:37:07.534 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:07.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.536+0000 7f72a9920640 1 Processor -- start 2024-03-29T11:37:07.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.537+0000 7f72a9920640 1 -- start start 2024-03-29T11:37:07.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.537+0000 7f72a9920640 10 monclient: init 2024-03-29T11:37:07.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.538+0000 7f72a9920640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:07.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.538+0000 7f72a9920640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:07.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.538+0000 7f72a9920640 10 monclient(hunting): picked mon.noname-c con 0x7f729c006890 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:07.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.538+0000 7f72a9920640 10 monclient(hunting): picked mon.noname-b con 0x7f729c006c70 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:07.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.538+0000 7f72a9920640 10 monclient(hunting): picked mon.noname-a con 0x7f729c02b4c0 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:07.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.542+0000 7f72a9920640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c02c830 con 0x7f729c02b4c0 2024-03-29T11:37:07.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.542+0000 7f72a9920640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c02afa0 con 0x7f729c006890 2024-03-29T11:37:07.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.542+0000 7f72a9920640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c02aba0 con 0x7f729c006c70 2024-03-29T11:37:07.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.542+0000 7f72a9920640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:07.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.542+0000 7f72a9920640 10 monclient(hunting): authenticate will time out at 1006.991703s 2024-03-29T11:37:07.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.542+0000 7f72a911f640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f729c02b4c0 0x7f729c0cf0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55367/0 (socket says 172.21.15.37:55367) 2024-03-29T11:37:07.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.542+0000 7f72a911f640 1 -- 172.21.15.37:0/1217666192 learned_addr learned my addr 172.21.15.37:0/1217666192 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:07.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.543+0000 7f72a37fe640 1 -- 172.21.15.37:0/1217666192 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497718398 0 0) 0x7f729c02c830 con 0x7f729c02b4c0 2024-03-29T11:37:07.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.543+0000 7f72a37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:07.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.543+0000 7f72a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:07.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.543+0000 7f72a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:07.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.543+0000 7f72a37fe640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:37:07.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.543+0000 7f72a37fe640 1 -- 172.21.15.37:0/1217666192 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7294003520 con 0x7f729c02b4c0 2024-03-29T11:37:07.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 1 -- 172.21.15.37:0/1217666192 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 883129033 0 0) 0x7f7294003520 con 0x7f729c02b4c0 2024-03-29T11:37:07.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 1 -- 172.21.15.37:0/1217666192 >> v1:172.21.15.37:6790/0 conn(0x7f729c006c70 legacy=0x7f729c0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:07.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 1 -- 172.21.15.37:0/1217666192 >> v1:172.21.15.146:6789/0 conn(0x7f729c006890 legacy=0x7f729c0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:07.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:07.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 1 monclient: found mon.noname-a 2024-03-29T11:37:07.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:07.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:37:07.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 1 -- 172.21.15.37:0/1217666192 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c02ad30 con 0x7f729c02b4c0 2024-03-29T11:37:07.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:07.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:07.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:07.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 1 -- 172.21.15.37:0/1217666192 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f72a404f730 con 0x7f729c02b4c0 2024-03-29T11:37:07.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 10 monclient: dump: 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a9920640 5 monclient: authenticate success, global_id 4293 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a9920640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:07.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 1 -- 172.21.15.37:0/1217666192 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72a4075f10 con 0x7f729c02b4c0 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 1 -- 172.21.15.37:0/1217666192 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f72a40758b0 con 0x7f729c02b4c0 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a37fe640 10 monclient: dump: 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:07.553 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:07.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a9920640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:07.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.548+0000 7f72a9920640 4 set_mon_vals no callback set 2024-03-29T11:37:07.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.550+0000 7f72a9920640 10 monclient: shutdown 2024-03-29T11:37:07.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.550+0000 7f72a9920640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:07.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.550+0000 7f72a9920640 1 -- 172.21.15.37:0/1217666192 >> v1:172.21.15.37:6789/0 conn(0x7f729c02b4c0 legacy=0x7f729c0cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:07.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.550+0000 7f72a9920640 1 -- 172.21.15.37:0/1217666192 shutdown_connections 2024-03-29T11:37:07.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.550+0000 7f72a9920640 1 -- 172.21.15.37:0/1217666192 >> 172.21.15.37:0/1217666192 conn(0x7f729c0058f0 msgr2=0x7f729c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:07.555 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.550+0000 7f72a9920640 1 -- 172.21.15.37:0/1217666192 shutdown_connections 2024-03-29T11:37:07.555 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.553+0000 7f72a9920640 1 -- 172.21.15.37:0/1217666192 wait complete. 2024-03-29T11:37:07.555 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.554+0000 7f72a9920640 10 monclient: build_initial_monmap 2024-03-29T11:37:07.555 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.554+0000 7f72a9920640 10 monclient: monmap: 2024-03-29T11:37:07.555 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:07.555 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.555+0000 7f72a9920640 1 librados: starting msgr at 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.555+0000 7f72a9920640 1 librados: starting objecter 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.555+0000 7f72a9920640 1 Processor -- start 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.555+0000 7f72a9920640 1 -- start start 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.555+0000 7f72a9920640 1 librados: setting wanted keys 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.555+0000 7f72a9920640 1 librados: calling monclient init 2024-03-29T11:37:07.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.555+0000 7f72a9920640 10 monclient: init 2024-03-29T11:37:07.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.556+0000 7f72a9920640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:07.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.557+0000 7f72a9920640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:07.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.557+0000 7f72a9920640 10 monclient(hunting): picked mon.noname-c con 0x7f729c006890 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:07.559 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.560+0000 7f72a9920640 10 monclient(hunting): picked mon.noname-a con 0x7f729c02b4c0 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:07.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.560+0000 7f72a9920640 10 monclient(hunting): picked mon.noname-b con 0x7f729c0c0780 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:07.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.560+0000 7f72a891e640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f729c006890 0x7f729c02e1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:55245/0 (socket says 172.21.15.37:55245) 2024-03-29T11:37:07.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.561+0000 7f72a891e640 1 -- 172.21.15.37:0/1666674892 learned_addr learned my addr 172.21.15.37:0/1666674892 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:07.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.561+0000 7f72a9920640 1 -- 172.21.15.37:0/1666674892 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c031b10 con 0x7f729c02b4c0 2024-03-29T11:37:07.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.561+0000 7f72a9920640 1 -- 172.21.15.37:0/1666674892 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c0bba70 con 0x7f729c0c0780 2024-03-29T11:37:07.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.560+0000 7f72a3fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f729c02b4c0 0x7f729c02e8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55501/0 (socket says 172.21.15.37:55501) 2024-03-29T11:37:07.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.561+0000 7f72a9920640 1 -- 172.21.15.37:0/1666674892 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c0bbce0 con 0x7f729c006890 2024-03-29T11:37:07.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.561+0000 7f72a9920640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:07.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.561+0000 7f72a9920640 10 monclient(hunting): authenticate will time out at 1007.010232s 2024-03-29T11:37:07.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.561+0000 7f72a17fa640 10 client.?.objecter ms_handle_connect 0x7f729c02b4c0 2024-03-29T11:37:07.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.561+0000 7f72a17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:07.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.561+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801276536 0 0) 0x7f729c031b10 con 0x7f729c02b4c0 2024-03-29T11:37:07.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.561+0000 7f72a17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:07.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.561+0000 7f72a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:07.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.561+0000 7f72a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:07.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.562+0000 7f72a17fa640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:37:07.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.562+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7288004eb0 con 0x7f729c02b4c0 2024-03-29T11:37:07.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.562+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3313342672 0 0) 0x7f7288004eb0 con 0x7f729c02b4c0 2024-03-29T11:37:07.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.564+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f729c031b10 con 0x7f729c02b4c0 2024-03-29T11:37:07.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.564+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f7298001950 con 0x7f729c02b4c0 2024-03-29T11:37:07.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.564+0000 7f72a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:07.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.564+0000 7f72a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:07.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.564+0000 7f72a17fa640 10 monclient(hunting): dump: 2024-03-29T11:37:07.567 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:07.567 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:07.567 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:07.567 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:07.567 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:07.567 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:07.567 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.564+0000 7f72a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.564+0000 7f72a17fa640 10 client.?.objecter ms_handle_connect 0x7f729c0c0780 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.564+0000 7f72a17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.565+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828393079 0 0) 0x7f729c0bba70 con 0x7f729c0c0780 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.565+0000 7f72a17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.565+0000 7f72a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.565+0000 7f72a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.565+0000 7f72a17fa640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.565+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7288004eb0 con 0x7f729c0c0780 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.565+0000 7f72a17fa640 10 client.?.objecter ms_handle_connect 0x7f729c006890 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.565+0000 7f72a17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.565+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1559765990 0 0) 0x7f729c031b10 con 0x7f729c02b4c0 2024-03-29T11:37:07.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.565+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 >> v1:172.21.15.37:6790/0 conn(0x7f729c006890 legacy=0x7f729c02e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.565+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 >> v1:172.21.15.146:6789/0 conn(0x7f729c0c0780 legacy=0x7f729c167320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a17fa640 1 monclient: found mon.a 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c0badd0 con 0x7f729c02b4c0 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a17fa640 10 monclient: _finish_auth 0 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a9920640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a9920640 10 monclient: _renew_subs 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a9920640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a9920640 1 -- 172.21.15.37:0/1666674892 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f729c030da0 con 0x7f729c02b4c0 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a9920640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:07.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a9920640 10 monclient: _renew_subs 2024-03-29T11:37:07.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a9920640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:37:07.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a9920640 1 -- 172.21.15.37:0/1666674892 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f729c168a40 con 0x7f729c02b4c0 2024-03-29T11:37:07.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a9920640 1 librados: init done 2024-03-29T11:37:07.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7298001c50 con 0x7f729c02b4c0 2024-03-29T11:37:07.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:07.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f7298003a60 con 0x7f729c02b4c0 2024-03-29T11:37:07.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:07.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:07.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a17fa640 10 monclient: dump: 2024-03-29T11:37:07.570 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:07.571 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:07.571 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:07.571 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:07.571 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:07.571 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:07.571 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:07.571 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:07.571 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:07.571 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:07.571 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.566+0000 7f72a27fc640 4 set_mon_vals no callback set 2024-03-29T11:37:07.571 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.568+0000 7f72a9920640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:07.572 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.569+0000 7f72a9920640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:07.572 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.569+0000 7f72a9920640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:37:07.572 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.569+0000 7f72a9920640 1 -- 172.21.15.37:0/1666674892 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7284005080 con 0x7f729c02b4c0 2024-03-29T11:37:07.572 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.569+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f7298003bc0 con 0x7f729c02b4c0 2024-03-29T11:37:07.572 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.570+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f7298084a70 con 0x7f729c02b4c0 2024-03-29T11:37:07.572 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.570+0000 7f72a17fa640 10 client.4296.objecter ms_dispatch 0x7f72a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:07.572 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.570+0000 7f72a17fa640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:07.572 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.570+0000 7f72a17fa640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:07.572 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.570+0000 7f72a17fa640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-29T11:37:07.572 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.571+0000 7f72a891e640 10 monclient: get_auth_request con 0x7f728806d9e0 auth_method 0 2024-03-29T11:37:07.574 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.572+0000 7f72a17fa640 10 client.4296.objecter ms_handle_connect 0x7f728806d9e0 2024-03-29T11:37:07.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.579+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7298050350 con 0x7f729c02b4c0 2024-03-29T11:37:07.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.579+0000 7f72a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:07.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.579+0000 7f72a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:07.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.914+0000 7f72a9920640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:07.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.914+0000 7f72a9920640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:07.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.914+0000 7f72a9920640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:37:07.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.914+0000 7f72a9920640 1 -- 172.21.15.37:0/1666674892 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7284005330 con 0x7f729c02b4c0 2024-03-29T11:37:07.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.915+0000 7f72a17fa640 1 -- 172.21.15.37:0/1666674892 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f7298052280 con 0x7f729c02b4c0 2024-03-29T11:37:07.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.915+0000 7f72a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:07.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.915+0000 7f72a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:07.917 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:37:07.917 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:07.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.918+0000 7f72a9920640 10 librados: watch_flush enter 2024-03-29T11:37:07.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.918+0000 7f72a9920640 10 librados: watch_flush exit 2024-03-29T11:37:07.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.918+0000 7f72a9920640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-29T11:37:07.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.918+0000 7f72a9920640 10 client.4296.objecter successfully canceled tick 2024-03-29T11:37:07.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.919+0000 7f72a9920640 1 -- 172.21.15.37:0/1666674892 >> v1:172.21.15.146:6816/2899149139 conn(0x7f728806d9e0 legacy=0x7f728806fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:07.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.919+0000 7f72a9920640 10 monclient: shutdown 2024-03-29T11:37:07.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.919+0000 7f72a9920640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:07.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.919+0000 7f72a9920640 1 -- 172.21.15.37:0/1666674892 >> v1:172.21.15.37:6789/0 conn(0x7f729c02b4c0 legacy=0x7f729c02e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:07.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.919+0000 7f72a9920640 1 -- 172.21.15.37:0/1666674892 shutdown_connections 2024-03-29T11:37:07.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.919+0000 7f72a9920640 1 -- 172.21.15.37:0/1666674892 >> 172.21.15.37:0/1666674892 conn(0x7f729c0058f0 msgr2=0x7f729c038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:07.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.919+0000 7f72a9920640 1 -- 172.21.15.37:0/1666674892 shutdown_connections 2024-03-29T11:37:07.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.920+0000 7f72a9920640 1 -- 172.21.15.37:0/1666674892 wait complete. 2024-03-29T11:37:07.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:07.920+0000 7f72a9920640 1 librados: shutdown 2024-03-29T11:37:07.936 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:14.743 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:14.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.898+0000 7f90dcd4d640 10 monclient: get_monmap_and_config 2024-03-29T11:37:14.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.898+0000 7f90dcd4d640 10 monclient: build_initial_monmap 2024-03-29T11:37:14.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.898+0000 7f90dcd4d640 10 monclient: monmap: 2024-03-29T11:37:14.898 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:14.898 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:14.898 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:37:14.899422+0000 2024-03-29T11:37:14.898 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:37:14.899422+0000 2024-03-29T11:37:14.898 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:14.898 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:14.898 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:14.899 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:37:14.899 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:37:14.899 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:14.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.899+0000 7f90dcd4d640 1 Processor -- start 2024-03-29T11:37:14.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.899+0000 7f90dcd4d640 1 -- start start 2024-03-29T11:37:14.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.899+0000 7f90dcd4d640 10 monclient: init 2024-03-29T11:37:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.901+0000 7f90dcd4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.901+0000 7f90dcd4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.901+0000 7f90dcd4d640 10 monclient(hunting): picked mon.noname-b con 0x7f90d81457f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.901+0000 7f90dcd4d640 10 monclient(hunting): picked mon.noname-c con 0x7f90d813f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.901+0000 7f90dcd4d640 10 monclient(hunting): picked mon.noname-a con 0x7f90d80a9d70 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.901+0000 7f90dcd4d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d80aaef0 con 0x7f90d80a9d70 2024-03-29T11:37:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.901+0000 7f90dcd4d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d8145bd0 con 0x7f90d813f650 2024-03-29T11:37:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.901+0000 7f90dcd4d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d8145eb0 con 0x7f90d81457f0 2024-03-29T11:37:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.901+0000 7f90dcd4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.901+0000 7f90dcd4d640 10 monclient(hunting): authenticate will time out at 1014.350212s 2024-03-29T11:37:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.901+0000 7f90d659c640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f90d81457f0 0x7f90d813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:59718/0 (socket says 172.21.15.37:59718) 2024-03-29T11:37:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.901+0000 7f90d659c640 1 -- 172.21.15.37:0/1085904414 learned_addr learned my addr 172.21.15.37:0/1085904414 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:14.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 1 -- 172.21.15.37:0/1085904414 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909431224 0 0) 0x7f90d80aaef0 con 0x7f90d80a9d70 2024-03-29T11:37:14.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient(hunting): _negotiate 2024-03-29T11:37:14.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:14.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:14.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:37:14.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 1 -- 172.21.15.37:0/1085904414 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c4003520 con 0x7f90d80a9d70 2024-03-29T11:37:14.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 1 -- 172.21.15.37:0/1085904414 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091953301 0 0) 0x7f90d8145eb0 con 0x7f90d81457f0 2024-03-29T11:37:14.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient(hunting): _negotiate 2024-03-29T11:37:14.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:14.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:14.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:37:14.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 1 -- 172.21.15.37:0/1085904414 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d80aaef0 con 0x7f90d81457f0 2024-03-29T11:37:14.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 1 -- 172.21.15.37:0/1085904414 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1291357479 0 0) 0x7f90c4003520 con 0x7f90d80a9d70 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 1 -- 172.21.15.37:0/1085904414 >> v1:172.21.15.37:6790/0 conn(0x7f90d81457f0 legacy=0x7f90d813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 1 -- 172.21.15.37:0/1085904414 >> v1:172.21.15.146:6789/0 conn(0x7f90d813f650 legacy=0x7f90d80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient: _finish_hunting 0 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 1 monclient: found mon.noname-a 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 1 -- 172.21.15.37:0/1085904414 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d8146340 con 0x7f90d80a9d70 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient: _finish_auth 0 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient: _check_auth_tickets 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 1 -- 172.21.15.37:0/1085904414 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f90d0001a10 con 0x7f90d80a9d70 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90d559a640 10 monclient: dump: 2024-03-29T11:37:14.904 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90dcd4d640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.902+0000 7f90dcd4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.903+0000 7f90d559a640 1 -- 172.21.15.37:0/1085904414 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90d0001b70 con 0x7f90d80a9d70 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.903+0000 7f90d559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.903+0000 7f90d559a640 1 -- 172.21.15.37:0/1085904414 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f90d0001f80 con 0x7f90d80a9d70 2024-03-29T11:37:14.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.903+0000 7f90d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:14.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.903+0000 7f90d559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:14.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.903+0000 7f90d559a640 10 monclient: dump: 2024-03-29T11:37:14.906 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:14.906 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:14.906 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:14.906 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:14.906 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:14.906 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:14.906 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:14.906 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:14.907 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:14.907 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:14.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.903+0000 7f90dcd4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:14.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.903+0000 7f90dcd4d640 4 set_mon_vals no callback set 2024-03-29T11:37:14.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.904+0000 7f90dcd4d640 10 monclient: shutdown 2024-03-29T11:37:14.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.904+0000 7f90dcd4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:14.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.904+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1085904414 >> v1:172.21.15.37:6789/0 conn(0x7f90d80a9d70 legacy=0x7f90d814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:14.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.904+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1085904414 shutdown_connections 2024-03-29T11:37:14.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.904+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1085904414 >> 172.21.15.37:0/1085904414 conn(0x7f90d8066a00 msgr2=0x7f90d8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:14.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.904+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1085904414 shutdown_connections 2024-03-29T11:37:14.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.904+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1085904414 wait complete. 2024-03-29T11:37:14.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.905+0000 7f90dcd4d640 10 monclient: build_initial_monmap 2024-03-29T11:37:14.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.905+0000 7f90dcd4d640 10 monclient: monmap: 2024-03-29T11:37:14.910 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:14.910 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:14.910 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.906+0000 7f90dcd4d640 1 librados: starting msgr at 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.906+0000 7f90dcd4d640 1 librados: starting objecter 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.906+0000 7f90dcd4d640 1 Processor -- start 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.906+0000 7f90dcd4d640 1 -- start start 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.906+0000 7f90dcd4d640 1 librados: setting wanted keys 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.906+0000 7f90dcd4d640 1 librados: calling monclient init 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.906+0000 7f90dcd4d640 10 monclient: init 2024-03-29T11:37:14.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.907+0000 7f90dcd4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:14.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.907+0000 7f90dcd4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:14.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.907+0000 7f90dcd4d640 10 monclient(hunting): picked mon.noname-a con 0x7f90d80a9d70 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:14.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.907+0000 7f90dcd4d640 10 monclient(hunting): picked mon.noname-b con 0x7f90d813f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:14.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.907+0000 7f90dcd4d640 10 monclient(hunting): picked mon.noname-c con 0x7f90d81457f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:14.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.907+0000 7f90dcd4d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d813ae80 con 0x7f90d80a9d70 2024-03-29T11:37:14.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.907+0000 7f90dcd4d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d806e210 con 0x7f90d813f650 2024-03-29T11:37:14.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.907+0000 7f90dcd4d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d806e4f0 con 0x7f90d81457f0 2024-03-29T11:37:14.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.907+0000 7f90dcd4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:14.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.907+0000 7f90dcd4d640 10 monclient(hunting): authenticate will time out at 1014.356753s 2024-03-29T11:37:14.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.908+0000 7f90d659c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f90d80a9d70 0x7f90d813a770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42056/0 (socket says 172.21.15.37:42056) 2024-03-29T11:37:14.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.908+0000 7f90d659c640 1 -- 172.21.15.37:0/1697821107 learned_addr learned my addr 172.21.15.37:0/1697821107 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:14.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.908+0000 7f90ceffd640 10 client.?.objecter ms_handle_connect 0x7f90d80a9d70 2024-03-29T11:37:14.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.908+0000 7f90ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:14.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.908+0000 7f90ceffd640 10 client.?.objecter ms_handle_connect 0x7f90d813f650 2024-03-29T11:37:14.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.908+0000 7f90ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:14.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.908+0000 7f90ceffd640 10 client.?.objecter ms_handle_connect 0x7f90d81457f0 2024-03-29T11:37:14.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.908+0000 7f90ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:14.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.908+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795790363 0 0) 0x7f90d813ae80 con 0x7f90d80a9d70 2024-03-29T11:37:14.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.908+0000 7f90ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:14.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.908+0000 7f90ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:14.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.908+0000 7f90ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:14.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.908+0000 7f90ceffd640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:37:14.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.908+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b8004eb0 con 0x7f90d80a9d70 2024-03-29T11:37:14.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874850492 0 0) 0x7f90d806e4f0 con 0x7f90d81457f0 2024-03-29T11:37:14.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:14.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b80051d0 con 0x7f90d81457f0 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 196322440 0 0) 0x7f90d806e210 con 0x7f90d813f650 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d806e4f0 con 0x7f90d813f650 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1465076624 0 0) 0x7f90b8004eb0 con 0x7f90d80a9d70 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90d806e210 con 0x7f90d80a9d70 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3927798246 0 0) 0x7f90b80051d0 con 0x7f90d81457f0 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90b8004eb0 con 0x7f90d81457f0 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f90c0001910 con 0x7f90d80a9d70 2024-03-29T11:37:14.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): dump: 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f90d0001e80 con 0x7f90d81457f0 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:14.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): dump: 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4140347450 0 0) 0x7f90d806e4f0 con 0x7f90d813f650 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90b80051d0 con 0x7f90d813f650 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f90c8001870 con 0x7f90d813f650 2024-03-29T11:37:14.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:14.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:14.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.909+0000 7f90ceffd640 10 monclient(hunting): dump: 2024-03-29T11:37:14.917 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:14.917 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:14.917 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:14.917 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:14.917 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:14.917 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:14.917 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:14.917 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:14.917 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:14.917 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3851645673 0 0) 0x7f90b8004eb0 con 0x7f90d81457f0 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 >> v1:172.21.15.146:6789/0 conn(0x7f90d813f650 legacy=0x7f90d8141780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 >> v1:172.21.15.37:6789/0 conn(0x7f90d80a9d70 legacy=0x7f90d813a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 1 monclient: found mon.c 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d813bbc0 con 0x7f90d81457f0 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 10 monclient: _finish_auth 0 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90dcd4d640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90dcd4d640 10 monclient: _renew_subs 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90dcd4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:14.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1697821107 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d81efb80 con 0x7f90d81457f0 2024-03-29T11:37:14.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90dcd4d640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:14.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90dcd4d640 10 monclient: _renew_subs 2024-03-29T11:37:14.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90dcd4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:14.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1697821107 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d81efef0 con 0x7f90d81457f0 2024-03-29T11:37:14.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90dcd4d640 1 librados: init done 2024-03-29T11:37:14.920 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90d00013e0 con 0x7f90d81457f0 2024-03-29T11:37:14.920 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:14.920 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f90d0003dc0 con 0x7f90d81457f0 2024-03-29T11:37:14.920 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90ceffd640 10 monclient: dump: 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.910+0000 7f90d4d99640 4 set_mon_vals no callback set 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.911+0000 7f90dcd4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.911+0000 7f90dcd4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:14.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.911+0000 7f90dcd4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:14.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.911+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1697821107 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90bc005080 con 0x7f90d81457f0 2024-03-29T11:37:14.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.913+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f90d0019d90 con 0x7f90d81457f0 2024-03-29T11:37:14.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.914+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f90d0083ac0 con 0x7f90d81457f0 2024-03-29T11:37:14.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.914+0000 7f90ceffd640 10 client.4316.objecter ms_dispatch 0x7f90d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:14.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.914+0000 7f90ceffd640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:14.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.914+0000 7f90ceffd640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:14.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.914+0000 7f90ceffd640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T11:37:14.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.914+0000 7f90d659c640 10 monclient: get_auth_request con 0x7f90b806e400 auth_method 0 2024-03-29T11:37:14.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.914+0000 7f90ceffd640 10 client.4316.objecter ms_handle_connect 0x7f90b806e400 2024-03-29T11:37:14.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.923+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90d0085e60 con 0x7f90d81457f0 2024-03-29T11:37:14.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.923+0000 7f90ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:14.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:14.923+0000 7f90ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:15.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.298+0000 7f90dcd4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:15.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.298+0000 7f90dcd4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:15.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.298+0000 7f90dcd4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:15.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.298+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1697821107 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90bc005330 con 0x7f90d81457f0 2024-03-29T11:37:15.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.300+0000 7f90ceffd640 1 -- 172.21.15.37:0/1697821107 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f90d004f3a0 con 0x7f90d81457f0 2024-03-29T11:37:15.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.300+0000 7f90ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:15.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.300+0000 7f90ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:15.299 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:37:15.299 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:15.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.303+0000 7f90dcd4d640 10 librados: watch_flush enter 2024-03-29T11:37:15.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.303+0000 7f90dcd4d640 10 librados: watch_flush exit 2024-03-29T11:37:15.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.303+0000 7f90dcd4d640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T11:37:15.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.303+0000 7f90dcd4d640 10 client.4316.objecter successfully canceled tick 2024-03-29T11:37:15.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.303+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1697821107 >> v1:172.21.15.146:6816/2899149139 conn(0x7f90b806e400 legacy=0x7f90b80708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:15.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.303+0000 7f90dcd4d640 10 monclient: shutdown 2024-03-29T11:37:15.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.303+0000 7f90dcd4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:15.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.303+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1697821107 >> v1:172.21.15.37:6790/0 conn(0x7f90d81457f0 legacy=0x7f90d8141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:15.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.304+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1697821107 shutdown_connections 2024-03-29T11:37:15.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.304+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1697821107 >> 172.21.15.37:0/1697821107 conn(0x7f90d8066a00 msgr2=0x7f90d8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:15.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.304+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1697821107 shutdown_connections 2024-03-29T11:37:15.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.304+0000 7f90dcd4d640 1 -- 172.21.15.37:0/1697821107 wait complete. 2024-03-29T11:37:15.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:15.304+0000 7f90dcd4d640 1 librados: shutdown 2024-03-29T11:37:15.318 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:22.125 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:22.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.266+0000 7fc3998f6640 10 monclient: get_monmap_and_config 2024-03-29T11:37:22.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.266+0000 7fc3998f6640 10 monclient: build_initial_monmap 2024-03-29T11:37:22.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.266+0000 7fc3998f6640 10 monclient: monmap: 2024-03-29T11:37:22.267 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:22.267 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:22.267 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:37:22.267766+0000 2024-03-29T11:37:22.267 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:37:22.267766+0000 2024-03-29T11:37:22.267 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:22.267 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:22.267 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:22.267 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:37:22.267 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:37:22.268 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:22.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.267+0000 7fc3998f6640 1 Processor -- start 2024-03-29T11:37:22.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.267+0000 7fc3998f6640 1 -- start start 2024-03-29T11:37:22.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.267+0000 7fc3998f6640 10 monclient: init 2024-03-29T11:37:22.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.268+0000 7fc3998f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:22.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.268+0000 7fc3998f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:22.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.268+0000 7fc3998f6640 10 monclient(hunting): picked mon.noname-b con 0x7fc3941457f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:22.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc3998f6640 10 monclient(hunting): picked mon.noname-c con 0x7fc39413f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:22.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc3998f6640 10 monclient(hunting): picked mon.noname-a con 0x7fc3940a9d70 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:22.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc3998f6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3940aaef0 con 0x7fc3940a9d70 2024-03-29T11:37:22.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc3998f6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc394145bd0 con 0x7fc39413f650 2024-03-29T11:37:22.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc3998f6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc394145eb0 con 0x7fc3941457f0 2024-03-29T11:37:22.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc3937fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc3940a9d70 0x7fc39414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47230/0 (socket says 172.21.15.37:47230) 2024-03-29T11:37:22.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc3937fe640 1 -- 172.21.15.37:0/1276268883 learned_addr learned my addr 172.21.15.37:0/1276268883 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:22.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc3998f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:22.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc3998f6640 10 monclient(hunting): authenticate will time out at 1021.718508s 2024-03-29T11:37:22.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc391ffb640 1 -- 172.21.15.37:0/1276268883 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594916038 0 0) 0x7fc3940aaef0 con 0x7fc3940a9d70 2024-03-29T11:37:22.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc391ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:22.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc391ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:22.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc391ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:22.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc391ffb640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:37:22.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.269+0000 7fc391ffb640 1 -- 172.21.15.37:0/1276268883 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc37c003520 con 0x7fc3940a9d70 2024-03-29T11:37:22.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.270+0000 7fc391ffb640 1 -- 172.21.15.37:0/1276268883 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024539986 0 0) 0x7fc394145eb0 con 0x7fc3941457f0 2024-03-29T11:37:22.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.270+0000 7fc391ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:22.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.270+0000 7fc391ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:22.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.270+0000 7fc391ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:22.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.270+0000 7fc391ffb640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:37:22.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.270+0000 7fc391ffb640 1 -- 172.21.15.37:0/1276268883 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3940aaef0 con 0x7fc3941457f0 2024-03-29T11:37:22.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.270+0000 7fc391ffb640 1 -- 172.21.15.37:0/1276268883 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4192848194 0 0) 0x7fc37c003520 con 0x7fc3940a9d70 2024-03-29T11:37:22.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.270+0000 7fc391ffb640 1 -- 172.21.15.37:0/1276268883 >> v1:172.21.15.37:6790/0 conn(0x7fc3941457f0 legacy=0x7fc39413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.270+0000 7fc391ffb640 1 -- 172.21.15.37:0/1276268883 >> v1:172.21.15.146:6789/0 conn(0x7fc39413f650 legacy=0x7fc3940b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:22.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.272+0000 7fc391ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:22.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.272+0000 7fc391ffb640 1 monclient: found mon.noname-a 2024-03-29T11:37:22.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.272+0000 7fc391ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.272+0000 7fc391ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.272+0000 7fc391ffb640 1 -- 172.21.15.37:0/1276268883 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc394146340 con 0x7fc3940a9d70 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc391ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc391ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc391ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc391ffb640 1 -- 172.21.15.37:0/1276268883 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc38c001a10 con 0x7fc3940a9d70 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc3998f6640 5 monclient: authenticate success, global_id 4305 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc3998f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc391ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc391ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc391ffb640 10 monclient: dump: 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:22.276 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc3998f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc391ffb640 1 -- 172.21.15.37:0/1276268883 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc38c001c30 con 0x7fc3940a9d70 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc391ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc391ffb640 1 -- 172.21.15.37:0/1276268883 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc38c001fc0 con 0x7fc3940a9d70 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc3998f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc3998f6640 4 set_mon_vals no callback set 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc391ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc391ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:22.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.273+0000 7fc391ffb640 10 monclient: dump: 2024-03-29T11:37:22.278 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:22.278 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:22.278 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:22.278 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:22.278 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:22.278 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:22.278 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:22.278 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:22.278 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:22.278 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:22.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.274+0000 7fc3998f6640 10 monclient: shutdown 2024-03-29T11:37:22.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.274+0000 7fc3998f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:22.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.274+0000 7fc3998f6640 1 -- 172.21.15.37:0/1276268883 >> v1:172.21.15.37:6789/0 conn(0x7fc3940a9d70 legacy=0x7fc39414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.274+0000 7fc3998f6640 1 -- 172.21.15.37:0/1276268883 shutdown_connections 2024-03-29T11:37:22.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.274+0000 7fc3998f6640 1 -- 172.21.15.37:0/1276268883 >> 172.21.15.37:0/1276268883 conn(0x7fc394066a00 msgr2=0x7fc394066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:22.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.275+0000 7fc3998f6640 1 -- 172.21.15.37:0/1276268883 shutdown_connections 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.275+0000 7fc3998f6640 1 -- 172.21.15.37:0/1276268883 wait complete. 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.276+0000 7fc3998f6640 10 monclient: build_initial_monmap 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.276+0000 7fc3998f6640 10 monclient: monmap: 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.276+0000 7fc3998f6640 1 librados: starting msgr at 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.276+0000 7fc3998f6640 1 librados: starting objecter 2024-03-29T11:37:22.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.277+0000 7fc3998f6640 1 Processor -- start 2024-03-29T11:37:22.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.277+0000 7fc3998f6640 1 -- start start 2024-03-29T11:37:22.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.277+0000 7fc3998f6640 1 librados: setting wanted keys 2024-03-29T11:37:22.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.277+0000 7fc3998f6640 1 librados: calling monclient init 2024-03-29T11:37:22.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.277+0000 7fc3998f6640 10 monclient: init 2024-03-29T11:37:22.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc3998f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:22.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc3998f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:22.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc3998f6640 10 monclient(hunting): picked mon.noname-a con 0x7fc3940a9d70 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:22.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc3998f6640 10 monclient(hunting): picked mon.noname-c con 0x7fc39413f650 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:22.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc3927fc640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fc39413f650 0x7fc3941d7620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:47412/0 (socket says 172.21.15.37:47412) 2024-03-29T11:37:22.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc3998f6640 10 monclient(hunting): picked mon.noname-b con 0x7fc3941457f0 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:22.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc3998f6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc394141e10 con 0x7fc3940a9d70 2024-03-29T11:37:22.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc3998f6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3940b0d40 con 0x7fc3941457f0 2024-03-29T11:37:22.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc3998f6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc394142060 con 0x7fc39413f650 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc3998f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc3998f6640 10 monclient(hunting): authenticate will time out at 1021.728406s 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc392ffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc3940a9d70 0x7fc3941427d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47530/0 (socket says 172.21.15.37:47530) 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc392ffd640 1 -- 172.21.15.37:0/2604103823 learned_addr learned my addr 172.21.15.37:0/2604103823 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc37bfff640 10 client.?.objecter ms_handle_connect 0x7fc39413f650 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc37bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677747944 0 0) 0x7fc394142060 con 0x7fc39413f650 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc37bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc37bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.279+0000 7fc37bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc380004eb0 con 0x7fc39413f650 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 10 client.?.objecter ms_handle_connect 0x7fc3940a9d70 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:22.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568980120 0 0) 0x7fc394141e10 con 0x7fc3940a9d70 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc394142060 con 0x7fc3940a9d70 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1596916263 0 0) 0x7fc380004eb0 con 0x7fc39413f650 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc394141e10 con 0x7fc39413f650 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc388001870 con 0x7fc39413f650 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 10 monclient(hunting): dump: 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4216164772 0 0) 0x7fc394142060 con 0x7fc3940a9d70 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc380004eb0 con 0x7fc3940a9d70 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc3840018d0 con 0x7fc3940a9d70 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.280+0000 7fc37bfff640 10 monclient(hunting): dump: 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:22.292 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:22.292 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:22.292 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:22.292 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:22.292 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:22.292 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:22.292 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:22.292 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:22.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.281+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2497535133 0 0) 0x7fc394141e10 con 0x7fc39413f650 2024-03-29T11:37:22.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.281+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 >> v1:172.21.15.146:6789/0 conn(0x7fc3941457f0 legacy=0x7fc3941d7d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:22.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.282+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 >> v1:172.21.15.37:6789/0 conn(0x7fc3940a9d70 legacy=0x7fc3941427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.282+0000 7fc37bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:22.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.283+0000 7fc37bfff640 1 monclient: found mon.c 2024-03-29T11:37:22.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.283+0000 7fc37bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:22.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.283+0000 7fc37bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:22.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.283+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39413b790 con 0x7fc39413f650 2024-03-29T11:37:22.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.283+0000 7fc37bfff640 10 monclient: _finish_auth 0 2024-03-29T11:37:22.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.283+0000 7fc37bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:22.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.283+0000 7fc37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:22.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.283+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc388001a50 con 0x7fc39413f650 2024-03-29T11:37:22.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.283+0000 7fc37bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:22.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.283+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc3880049e0 con 0x7fc39413f650 2024-03-29T11:37:22.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.283+0000 7fc3917fa640 4 set_mon_vals no callback set 2024-03-29T11:37:22.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.283+0000 7fc37bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:22.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.284+0000 7fc37bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:22.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.284+0000 7fc37bfff640 10 monclient: dump: 2024-03-29T11:37:22.294 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:22.294 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:22.294 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:22.294 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:22.294 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:22.294 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:22.294 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:22.294 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:22.294 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:22.294 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:22.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.284+0000 7fc3998f6640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:37:22.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.284+0000 7fc3998f6640 10 monclient: _renew_subs 2024-03-29T11:37:22.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.284+0000 7fc3998f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:22.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.284+0000 7fc3998f6640 1 -- 172.21.15.37:0/2604103823 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc39413b9e0 con 0x7fc39413f650 2024-03-29T11:37:22.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.284+0000 7fc3998f6640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:22.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.284+0000 7fc3998f6640 10 monclient: _renew_subs 2024-03-29T11:37:22.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.284+0000 7fc3998f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:22.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.284+0000 7fc3998f6640 1 -- 172.21.15.37:0/2604103823 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3941e30a0 con 0x7fc39413f650 2024-03-29T11:37:22.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.284+0000 7fc3998f6640 1 librados: init done 2024-03-29T11:37:22.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.285+0000 7fc3998f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:22.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.285+0000 7fc3998f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:22.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.285+0000 7fc3998f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:22.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.285+0000 7fc3998f6640 1 -- 172.21.15.37:0/2604103823 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc394075f10 con 0x7fc39413f650 2024-03-29T11:37:22.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.286+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fc38801a9b0 con 0x7fc39413f650 2024-03-29T11:37:22.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.286+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fc388084a40 con 0x7fc39413f650 2024-03-29T11:37:22.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.286+0000 7fc37bfff640 10 client.4322.objecter ms_dispatch 0x7fc394000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:22.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.286+0000 7fc37bfff640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:22.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.286+0000 7fc37bfff640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:22.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.287+0000 7fc37bfff640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-29T11:37:22.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.287+0000 7fc392ffd640 10 monclient: get_auth_request con 0x7fc38006de80 auth_method 0 2024-03-29T11:37:22.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.287+0000 7fc37bfff640 10 client.4322.objecter ms_handle_connect 0x7fc38006de80 2024-03-29T11:37:22.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.293+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc388050150 con 0x7fc39413f650 2024-03-29T11:37:22.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.293+0000 7fc37bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:22.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.293+0000 7fc37bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:22.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.616+0000 7fc3998f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:22.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.616+0000 7fc3998f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:22.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.616+0000 7fc3998f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:22.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.616+0000 7fc3998f6640 1 -- 172.21.15.37:0/2604103823 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3940b3680 con 0x7fc39413f650 2024-03-29T11:37:22.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.617+0000 7fc37bfff640 1 -- 172.21.15.37:0/2604103823 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fc388052250 con 0x7fc39413f650 2024-03-29T11:37:22.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.617+0000 7fc37bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:22.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.617+0000 7fc37bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:22.617 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:37:22.617 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:22.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.620+0000 7fc3998f6640 10 librados: watch_flush enter 2024-03-29T11:37:22.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.620+0000 7fc3998f6640 10 librados: watch_flush exit 2024-03-29T11:37:22.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.620+0000 7fc3998f6640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-29T11:37:22.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.620+0000 7fc3998f6640 10 client.4322.objecter successfully canceled tick 2024-03-29T11:37:22.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.620+0000 7fc3998f6640 1 -- 172.21.15.37:0/2604103823 >> v1:172.21.15.146:6816/2899149139 conn(0x7fc38006de80 legacy=0x7fc380070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.620+0000 7fc3998f6640 10 monclient: shutdown 2024-03-29T11:37:22.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.620+0000 7fc3998f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:22.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.620+0000 7fc3998f6640 1 -- 172.21.15.37:0/2604103823 >> v1:172.21.15.37:6790/0 conn(0x7fc39413f650 legacy=0x7fc3941d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.620+0000 7fc3998f6640 1 -- 172.21.15.37:0/2604103823 shutdown_connections 2024-03-29T11:37:22.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.620+0000 7fc3998f6640 1 -- 172.21.15.37:0/2604103823 >> 172.21.15.37:0/2604103823 conn(0x7fc394066a00 msgr2=0x7fc3940b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:22.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.620+0000 7fc3998f6640 1 -- 172.21.15.37:0/2604103823 shutdown_connections 2024-03-29T11:37:22.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.620+0000 7fc3998f6640 1 -- 172.21.15.37:0/2604103823 wait complete. 2024-03-29T11:37:22.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:22.621+0000 7fc3998f6640 1 librados: shutdown 2024-03-29T11:37:22.634 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:29.441 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:29.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.587+0000 7f7e89fcf640 10 monclient: get_monmap_and_config 2024-03-29T11:37:29.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.587+0000 7f7e89fcf640 10 monclient: build_initial_monmap 2024-03-29T11:37:29.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.587+0000 7f7e89fcf640 10 monclient: monmap: 2024-03-29T11:37:29.588 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:29.588 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:29.588 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:37:29.588828+0000 2024-03-29T11:37:29.588 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:37:29.588828+0000 2024-03-29T11:37:29.588 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:29.588 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:29.588 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:29.588 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:37:29.588 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:37:29.588 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:29.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.592+0000 7f7e89fcf640 1 Processor -- start 2024-03-29T11:37:29.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.592+0000 7f7e89fcf640 1 -- start start 2024-03-29T11:37:29.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.592+0000 7f7e89fcf640 10 monclient: init 2024-03-29T11:37:29.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.593+0000 7f7e89fcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:29.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.593+0000 7f7e89fcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:29.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.593+0000 7f7e89fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f7e840b0860 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:29.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.593+0000 7f7e89fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f7e84072450 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:29.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.593+0000 7f7e89fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f7e8413ee40 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:29.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.593+0000 7f7e89fcf640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e840ad840 con 0x7f7e8413ee40 2024-03-29T11:37:29.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.593+0000 7f7e89fcf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e8413c830 con 0x7f7e84072450 2024-03-29T11:37:29.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.593+0000 7f7e89fcf640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e8413c9c0 con 0x7f7e840b0860 2024-03-29T11:37:29.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.593+0000 7f7e89fcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:29.596 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.593+0000 7f7e89fcf640 10 monclient(hunting): authenticate will time out at 1029.042166s 2024-03-29T11:37:29.596 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.595+0000 7f7e82ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f7e84072450 0x7f7e8413c120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:46875/0 (socket says 172.21.15.37:46875) 2024-03-29T11:37:29.596 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.595+0000 7f7e82ffd640 1 -- 172.21.15.37:0/199785617 learned_addr learned my addr 172.21.15.37:0/199785617 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:29.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.596+0000 7f7e827fc640 1 -- 172.21.15.37:0/199785617 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536396705 0 0) 0x7f7e8413c830 con 0x7f7e84072450 2024-03-29T11:37:29.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.596+0000 7f7e827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:29.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.596+0000 7f7e827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:29.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.596+0000 7f7e827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:29.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.596+0000 7f7e827fc640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:37:29.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.596+0000 7f7e827fc640 1 -- 172.21.15.37:0/199785617 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e6c003520 con 0x7f7e84072450 2024-03-29T11:37:29.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.596+0000 7f7e827fc640 1 -- 172.21.15.37:0/199785617 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276133308 0 0) 0x7f7e840ad840 con 0x7f7e8413ee40 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.596+0000 7f7e827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.596+0000 7f7e827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.596+0000 7f7e827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.596+0000 7f7e827fc640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 1 -- 172.21.15.37:0/199785617 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e8413c830 con 0x7f7e8413ee40 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 1 -- 172.21.15.37:0/199785617 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291209116 0 0) 0x7f7e8413c9c0 con 0x7f7e840b0860 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 1 -- 172.21.15.37:0/199785617 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e840ad840 con 0x7f7e840b0860 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 1 -- 172.21.15.37:0/199785617 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2228530441 0 0) 0x7f7e6c003520 con 0x7f7e84072450 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 1 -- 172.21.15.37:0/199785617 >> v1:172.21.15.37:6790/0 conn(0x7f7e840b0860 legacy=0x7f7e840acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 1 -- 172.21.15.37:0/199785617 >> v1:172.21.15.37:6789/0 conn(0x7f7e8413ee40 legacy=0x7f7e8413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:29.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 1 monclient: found mon.noname-c 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 1 -- 172.21.15.37:0/199785617 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8413f970 con 0x7f7e84072450 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 1 -- 172.21.15.37:0/199785617 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f7e74001690 con 0x7f7e84072450 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient: dump: 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:29.599 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 1 -- 172.21.15.37:0/199785617 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e740018b0 con 0x7f7e84072450 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 1 -- 172.21.15.37:0/199785617 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f7e74001c40 con 0x7f7e84072450 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e827fc640 10 monclient: dump: 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:29.600 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:29.601 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:29.601 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:29.601 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:29.601 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:29.601 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:29.601 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:29.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e89fcf640 5 monclient: authenticate success, global_id 4315 2024-03-29T11:37:29.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e89fcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:29.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.597+0000 7f7e89fcf640 4 set_mon_vals no callback set 2024-03-29T11:37:29.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.598+0000 7f7e89fcf640 10 monclient: shutdown 2024-03-29T11:37:29.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.598+0000 7f7e89fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:29.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.598+0000 7f7e89fcf640 1 -- 172.21.15.37:0/199785617 >> v1:172.21.15.146:6789/0 conn(0x7f7e84072450 legacy=0x7f7e8413c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:29.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.598+0000 7f7e89fcf640 1 -- 172.21.15.37:0/199785617 shutdown_connections 2024-03-29T11:37:29.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.598+0000 7f7e89fcf640 1 -- 172.21.15.37:0/199785617 >> 172.21.15.37:0/199785617 conn(0x7f7e8406b660 msgr2=0x7f7e8406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:29.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.601+0000 7f7e89fcf640 1 -- 172.21.15.37:0/199785617 shutdown_connections 2024-03-29T11:37:29.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.601+0000 7f7e89fcf640 1 -- 172.21.15.37:0/199785617 wait complete. 2024-03-29T11:37:29.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.602+0000 7f7e89fcf640 10 monclient: build_initial_monmap 2024-03-29T11:37:29.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.602+0000 7f7e89fcf640 10 monclient: monmap: 2024-03-29T11:37:29.603 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:29.603 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:29.603 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:37:29.603 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:37:29.604 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:29.604 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:29.604 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:29.604 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:37:29.604 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:37:29.604 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:29.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.602+0000 7f7e89fcf640 1 librados: starting msgr at 2024-03-29T11:37:29.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.602+0000 7f7e89fcf640 1 librados: starting objecter 2024-03-29T11:37:29.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.602+0000 7f7e89fcf640 1 Processor -- start 2024-03-29T11:37:29.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.603+0000 7f7e89fcf640 1 -- start start 2024-03-29T11:37:29.608 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.603+0000 7f7e89fcf640 1 librados: setting wanted keys 2024-03-29T11:37:29.608 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.603+0000 7f7e89fcf640 1 librados: calling monclient init 2024-03-29T11:37:29.608 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.603+0000 7f7e89fcf640 10 monclient: init 2024-03-29T11:37:29.608 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e89fcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:29.608 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e89fcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:29.608 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e89fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f7e84072450 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e89fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f7e840b0860 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e89fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f7e8413ee40 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e89fcf640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e840b39d0 con 0x7f7e8413ee40 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e89fcf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e840b35d0 con 0x7f7e840b0860 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e89fcf640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e840b3760 con 0x7f7e84072450 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e89fcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e89fcf640 10 monclient(hunting): authenticate will time out at 1029.053124s 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e837fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f7e84072450 0x7f7e84132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:34996/0 (socket says 172.21.15.37:34996) 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e837fe640 1 -- 172.21.15.37:0/1461039039 learned_addr learned my addr 172.21.15.37:0/1461039039 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e6bfff640 10 client.?.objecter ms_handle_connect 0x7f7e84072450 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e6bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200736527 0 0) 0x7f7e840b3760 con 0x7f7e84072450 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e6bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:29.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e6bfff640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.604+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e7000d6c0 con 0x7f7e84072450 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.605+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3287650830 0 0) 0x7f7e7000d6c0 con 0x7f7e84072450 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.605+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e840b3760 con 0x7f7e84072450 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.605+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f7e78001930 con 0x7f7e84072450 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.605+0000 7f7e6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.605+0000 7f7e6bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.605+0000 7f7e6bfff640 10 monclient(hunting): dump: 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:29.610 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:29.611 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:29.611 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:29.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.605+0000 7f7e6bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:29.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.605+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2889994059 0 0) 0x7f7e840b3760 con 0x7f7e84072450 2024-03-29T11:37:29.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.605+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 >> v1:172.21.15.146:6789/0 conn(0x7f7e840b0860 legacy=0x7f7e841d7230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:29.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.607+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 >> v1:172.21.15.37:6789/0 conn(0x7f7e8413ee40 legacy=0x7f7e841d7940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:29.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.608+0000 7f7e6bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:29.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.608+0000 7f7e6bfff640 1 monclient: found mon.c 2024-03-29T11:37:29.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.608+0000 7f7e6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:29.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.608+0000 7f7e6bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:29.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.608+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e840b2cd0 con 0x7f7e84072450 2024-03-29T11:37:29.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.608+0000 7f7e6bfff640 10 monclient: _finish_auth 0 2024-03-29T11:37:29.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.608+0000 7f7e6bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:29.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.608+0000 7f7e6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:29.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.609+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e78001d60 con 0x7f7e84072450 2024-03-29T11:37:29.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.609+0000 7f7e6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:29.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.609+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f7e78003a60 con 0x7f7e84072450 2024-03-29T11:37:29.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.609+0000 7f7e6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:29.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.609+0000 7f7e6bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:29.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.609+0000 7f7e6bfff640 10 monclient: dump: 2024-03-29T11:37:29.614 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:29.614 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:29.614 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:29.614 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:29.614 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:29.614 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:29.614 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:29.614 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:29.614 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.610+0000 7f7e81ffb640 4 set_mon_vals no callback set 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.610+0000 7f7e89fcf640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.610+0000 7f7e89fcf640 10 monclient: _renew_subs 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.610+0000 7f7e89fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.610+0000 7f7e89fcf640 1 -- 172.21.15.37:0/1461039039 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e840b27f0 con 0x7f7e84072450 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.610+0000 7f7e89fcf640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.610+0000 7f7e89fcf640 10 monclient: _renew_subs 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.610+0000 7f7e89fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.610+0000 7f7e89fcf640 1 -- 172.21.15.37:0/1461039039 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e84137bd0 con 0x7f7e84072450 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.610+0000 7f7e89fcf640 1 librados: init done 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.612+0000 7f7e89fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.612+0000 7f7e89fcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.612+0000 7f7e89fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:29.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.612+0000 7f7e89fcf640 1 -- 172.21.15.37:0/1461039039 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e840b3c10 con 0x7f7e84072450 2024-03-29T11:37:29.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.615+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f7e78003bc0 con 0x7f7e84072450 2024-03-29T11:37:29.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.615+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f7e78084960 con 0x7f7e84072450 2024-03-29T11:37:29.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.615+0000 7f7e6bfff640 10 client.4328.objecter ms_dispatch 0x7f7e84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:29.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.615+0000 7f7e6bfff640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:29.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.615+0000 7f7e6bfff640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:29.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.616+0000 7f7e6bfff640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T11:37:29.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.616+0000 7f7e82ffd640 10 monclient: get_auth_request con 0x7f7e70076570 auth_method 0 2024-03-29T11:37:29.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.616+0000 7f7e6bfff640 10 client.4328.objecter ms_handle_connect 0x7f7e70076570 2024-03-29T11:37:29.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.623+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e78050240 con 0x7f7e84072450 2024-03-29T11:37:29.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.623+0000 7f7e6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:29.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.623+0000 7f7e6bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:29.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.947+0000 7f7e89fcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:29.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.947+0000 7f7e89fcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:29.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.947+0000 7f7e89fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:29.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.947+0000 7f7e89fcf640 1 -- 172.21.15.37:0/1461039039 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e840b2af0 con 0x7f7e84072450 2024-03-29T11:37:29.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.950+0000 7f7e6bfff640 1 -- 172.21.15.37:0/1461039039 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f7e78052170 con 0x7f7e84072450 2024-03-29T11:37:29.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.950+0000 7f7e6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:29.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.950+0000 7f7e6bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:29.951 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:37:29.951 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:29.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.954+0000 7f7e89fcf640 10 librados: watch_flush enter 2024-03-29T11:37:29.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.954+0000 7f7e89fcf640 10 librados: watch_flush exit 2024-03-29T11:37:29.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.954+0000 7f7e89fcf640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T11:37:29.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.955+0000 7f7e89fcf640 10 client.4328.objecter successfully canceled tick 2024-03-29T11:37:29.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.956+0000 7f7e89fcf640 1 -- 172.21.15.37:0/1461039039 >> v1:172.21.15.146:6816/2899149139 conn(0x7f7e70076570 legacy=0x7f7e70078a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:29.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.956+0000 7f7e89fcf640 10 monclient: shutdown 2024-03-29T11:37:29.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.956+0000 7f7e89fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:29.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.956+0000 7f7e89fcf640 1 -- 172.21.15.37:0/1461039039 >> v1:172.21.15.37:6790/0 conn(0x7f7e84072450 legacy=0x7f7e84132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:29.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.957+0000 7f7e89fcf640 1 -- 172.21.15.37:0/1461039039 shutdown_connections 2024-03-29T11:37:29.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.957+0000 7f7e89fcf640 1 -- 172.21.15.37:0/1461039039 >> 172.21.15.37:0/1461039039 conn(0x7f7e8406b660 msgr2=0x7f7e84135c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:29.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.959+0000 7f7e89fcf640 1 -- 172.21.15.37:0/1461039039 shutdown_connections 2024-03-29T11:37:29.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.959+0000 7f7e89fcf640 1 -- 172.21.15.37:0/1461039039 wait complete. 2024-03-29T11:37:29.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:29.960+0000 7f7e89fcf640 1 librados: shutdown 2024-03-29T11:37:29.977 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:36.784 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:36.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.931+0000 7f192d2e0640 10 monclient: get_monmap_and_config 2024-03-29T11:37:36.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.931+0000 7f192d2e0640 10 monclient: build_initial_monmap 2024-03-29T11:37:36.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.931+0000 7f192d2e0640 10 monclient: monmap: 2024-03-29T11:37:36.934 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:36.934 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:36.934 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:37:36.933598+0000 2024-03-29T11:37:36.934 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:37:36.933598+0000 2024-03-29T11:37:36.934 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:36.934 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:36.934 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:36.934 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:37:36.934 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:37:36.934 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:36.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.932+0000 7f192d2e0640 1 Processor -- start 2024-03-29T11:37:36.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.932+0000 7f192d2e0640 1 -- start start 2024-03-29T11:37:36.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.932+0000 7f192d2e0640 10 monclient: init 2024-03-29T11:37:36.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.933+0000 7f192d2e0640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:36.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.933+0000 7f192d2e0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:36.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.933+0000 7f192d2e0640 10 monclient(hunting): picked mon.noname-a con 0x7f19280a9ac0 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:36.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.933+0000 7f192d2e0640 10 monclient(hunting): picked mon.noname-c con 0x7f192813ee90 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:36.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.933+0000 7f192d2e0640 10 monclient(hunting): picked mon.noname-b con 0x7f19280af900 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:36.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.933+0000 7f192d2e0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19280afd10 con 0x7f19280a9ac0 2024-03-29T11:37:36.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.933+0000 7f192d2e0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19280af6f0 con 0x7f192813ee90 2024-03-29T11:37:36.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.933+0000 7f192d2e0640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19280aac80 con 0x7f19280af900 2024-03-29T11:37:36.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.933+0000 7f192d2e0640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:36.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.933+0000 7f192d2e0640 10 monclient(hunting): authenticate will time out at 1036.383673s 2024-03-29T11:37:36.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.937+0000 7f1926ffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f19280a9ac0 0x7f19280adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:59981/0 (socket says 172.21.15.37:59981) 2024-03-29T11:37:36.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.937+0000 7f1926ffd640 1 -- 172.21.15.37:0/4091080626 learned_addr learned my addr 172.21.15.37:0/4091080626 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:36.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.937+0000 7f1925ffb640 1 -- 172.21.15.37:0/4091080626 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370741724 0 0) 0x7f19280afd10 con 0x7f19280a9ac0 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.937+0000 7f1925ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.937+0000 7f1925ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.937+0000 7f1925ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.937+0000 7f1925ffb640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.937+0000 7f1925ffb640 1 -- 172.21.15.37:0/4091080626 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1910003520 con 0x7f19280a9ac0 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 1 -- 172.21.15.37:0/4091080626 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 446339229 0 0) 0x7f1910003520 con 0x7f19280a9ac0 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 1 -- 172.21.15.37:0/4091080626 >> v1:172.21.15.37:6790/0 conn(0x7f19280af900 legacy=0x7f1928149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 1 -- 172.21.15.37:0/4091080626 >> v1:172.21.15.146:6789/0 conn(0x7f192813ee90 legacy=0x7f192813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 1 monclient: found mon.noname-a 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 1 -- 172.21.15.37:0/4091080626 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19280aaf60 con 0x7f19280a9ac0 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:36.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 1 -- 172.21.15.37:0/4091080626 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1918001a10 con 0x7f19280a9ac0 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 10 monclient: dump: 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:36.943 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:36.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f192d2e0640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:37:36.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f192d2e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:36.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 1 -- 172.21.15.37:0/4091080626 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1918001be0 con 0x7f19280a9ac0 2024-03-29T11:37:36.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:36.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f192d2e0640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:36.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f192d2e0640 4 set_mon_vals no callback set 2024-03-29T11:37:36.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 1 -- 172.21.15.37:0/4091080626 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1918002030 con 0x7f19280a9ac0 2024-03-29T11:37:36.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:36.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:36.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.938+0000 7f1925ffb640 10 monclient: dump: 2024-03-29T11:37:36.944 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:36.944 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:36.944 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:36.944 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:36.945 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:36.945 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:36.945 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:36.945 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:36.945 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:36.945 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:36.945 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.939+0000 7f192d2e0640 10 monclient: shutdown 2024-03-29T11:37:36.945 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.939+0000 7f192d2e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:36.945 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.939+0000 7f192d2e0640 1 -- 172.21.15.37:0/4091080626 >> v1:172.21.15.37:6789/0 conn(0x7f19280a9ac0 legacy=0x7f19280adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:36.945 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.939+0000 7f192d2e0640 1 -- 172.21.15.37:0/4091080626 shutdown_connections 2024-03-29T11:37:36.945 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.939+0000 7f192d2e0640 1 -- 172.21.15.37:0/4091080626 >> 172.21.15.37:0/4091080626 conn(0x7f19280b35e0 msgr2=0x7f19280b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:36.945 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.940+0000 7f192d2e0640 1 -- 172.21.15.37:0/4091080626 shutdown_connections 2024-03-29T11:37:36.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.940+0000 7f192d2e0640 1 -- 172.21.15.37:0/4091080626 wait complete. 2024-03-29T11:37:36.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.940+0000 7f192d2e0640 10 monclient: build_initial_monmap 2024-03-29T11:37:36.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.940+0000 7f192d2e0640 10 monclient: monmap: 2024-03-29T11:37:36.951 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:36.951 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:36.952 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:37:36.952 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:37:36.952 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:36.952 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:36.952 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:36.952 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:37:36.952 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:37:36.952 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:36.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.941+0000 7f192d2e0640 1 librados: starting msgr at 2024-03-29T11:37:36.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.941+0000 7f192d2e0640 1 librados: starting objecter 2024-03-29T11:37:36.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.941+0000 7f192d2e0640 1 Processor -- start 2024-03-29T11:37:36.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f192d2e0640 1 -- start start 2024-03-29T11:37:36.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f192d2e0640 1 librados: setting wanted keys 2024-03-29T11:37:36.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f192d2e0640 1 librados: calling monclient init 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f192d2e0640 10 monclient: init 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f192d2e0640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f192d2e0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f192d2e0640 10 monclient(hunting): picked mon.noname-a con 0x7f19280a9ac0 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f192d2e0640 10 monclient(hunting): picked mon.noname-c con 0x7f19280af900 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f192d2e0640 10 monclient(hunting): picked mon.noname-b con 0x7f192813ee90 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f192d2e0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1928142330 con 0x7f19280a9ac0 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f192d2e0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19281db690 con 0x7f192813ee90 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f192d2e0640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19280b11d0 con 0x7f19280af900 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f192d2e0640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f192d2e0640 10 monclient(hunting): authenticate will time out at 1036.392864s 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f19267fc640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f19280af900 0x7f19281dbbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:51552/0 (socket says 172.21.15.37:51552) 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.942+0000 7f19267fc640 1 -- 172.21.15.37:0/3760804418 learned_addr learned my addr 172.21.15.37:0/3760804418 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.943+0000 7f190f7fe640 10 client.?.objecter ms_handle_connect 0x7f19280af900 2024-03-29T11:37:36.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.943+0000 7f190f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:36.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823898374 0 0) 0x7f19280b11d0 con 0x7f19280af900 2024-03-29T11:37:36.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:36.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:36.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:36.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:37:36.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1914004eb0 con 0x7f19280af900 2024-03-29T11:37:36.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3162095144 0 0) 0x7f1914004eb0 con 0x7f19280af900 2024-03-29T11:37:36.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19280b11d0 con 0x7f19280af900 2024-03-29T11:37:36.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f191c001990 con 0x7f19280af900 2024-03-29T11:37:36.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:36.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:36.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 10 monclient(hunting): dump: 2024-03-29T11:37:36.954 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:36.954 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:36.955 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:36.955 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:36.955 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:36.955 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:36.955 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:36.955 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:36.955 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:36.955 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:36.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:36.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4122160393 0 0) 0x7f19280b11d0 con 0x7f19280af900 2024-03-29T11:37:36.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 >> v1:172.21.15.146:6789/0 conn(0x7f192813ee90 legacy=0x7f1928071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:36.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.945+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 >> v1:172.21.15.37:6789/0 conn(0x7f19280a9ac0 legacy=0x7f192813fed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:36.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f190f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f190f7fe640 1 monclient: found mon.c 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f190f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f190f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19280b1360 con 0x7f19280af900 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f190f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f190f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f190f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f192d2e0640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f192d2e0640 10 monclient: _renew_subs 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f192d2e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f192d2e0640 1 -- 172.21.15.37:0/3760804418 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19281dc2c0 con 0x7f19280af900 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f192d2e0640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f192d2e0640 10 monclient: _renew_subs 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f192d2e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:36.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f192d2e0640 1 -- 172.21.15.37:0/3760804418 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1928072480 con 0x7f19280af900 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.947+0000 7f192d2e0640 1 librados: init done 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.950+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f191c001c90 con 0x7f19280af900 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.950+0000 7f190f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.950+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f191c0038c0 con 0x7f19280af900 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.950+0000 7f190f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.950+0000 7f190f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.950+0000 7f190f7fe640 10 monclient: dump: 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:36.957 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:36.958 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:36.958 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:36.958 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.950+0000 7f19257fa640 4 set_mon_vals no callback set 2024-03-29T11:37:36.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.953+0000 7f192d2e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:36.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.953+0000 7f192d2e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:36.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.953+0000 7f192d2e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:36.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.953+0000 7f192d2e0640 1 -- 172.21.15.37:0/3760804418 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1908005080 con 0x7f19280af900 2024-03-29T11:37:36.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.953+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f191c003a20 con 0x7f19280af900 2024-03-29T11:37:36.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.954+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f191c0849a0 con 0x7f19280af900 2024-03-29T11:37:36.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.954+0000 7f190f7fe640 10 client.4331.objecter ms_dispatch 0x7f1928000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:36.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.954+0000 7f190f7fe640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:36.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.954+0000 7f190f7fe640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:36.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.954+0000 7f190f7fe640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T11:37:36.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.955+0000 7f1926ffd640 10 monclient: get_auth_request con 0x7f191406dc70 auth_method 0 2024-03-29T11:37:36.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.955+0000 7f190f7fe640 10 client.4331.objecter ms_handle_connect 0x7f191406dc70 2024-03-29T11:37:36.963 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.962+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f191c050280 con 0x7f19280af900 2024-03-29T11:37:36.963 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.962+0000 7f190f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:36.963 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:36.962+0000 7f190f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:37.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.293+0000 7f192d2e0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:37.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.293+0000 7f192d2e0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:37.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.293+0000 7f192d2e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:37.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.293+0000 7f192d2e0640 1 -- 172.21.15.37:0/3760804418 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1908005330 con 0x7f19280af900 2024-03-29T11:37:37.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.294+0000 7f190f7fe640 1 -- 172.21.15.37:0/3760804418 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f191c0521b0 con 0x7f19280af900 2024-03-29T11:37:37.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.294+0000 7f190f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:37.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.294+0000 7f190f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:37.297 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:37:37.297 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:37.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.298+0000 7f192d2e0640 10 librados: watch_flush enter 2024-03-29T11:37:37.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.298+0000 7f192d2e0640 10 librados: watch_flush exit 2024-03-29T11:37:37.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.298+0000 7f192d2e0640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T11:37:37.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.298+0000 7f192d2e0640 10 client.4331.objecter successfully canceled tick 2024-03-29T11:37:37.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.298+0000 7f192d2e0640 1 -- 172.21.15.37:0/3760804418 >> v1:172.21.15.146:6816/2899149139 conn(0x7f191406dc70 legacy=0x7f1914070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.298+0000 7f192d2e0640 10 monclient: shutdown 2024-03-29T11:37:37.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.298+0000 7f192d2e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:37.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.298+0000 7f192d2e0640 1 -- 172.21.15.37:0/3760804418 >> v1:172.21.15.37:6790/0 conn(0x7f19280af900 legacy=0x7f19281dbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.298+0000 7f192d2e0640 1 -- 172.21.15.37:0/3760804418 shutdown_connections 2024-03-29T11:37:37.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.298+0000 7f192d2e0640 1 -- 172.21.15.37:0/3760804418 >> 172.21.15.37:0/3760804418 conn(0x7f19280b35e0 msgr2=0x7f192813b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:37.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.299+0000 7f192d2e0640 1 -- 172.21.15.37:0/3760804418 shutdown_connections 2024-03-29T11:37:37.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.299+0000 7f192d2e0640 1 -- 172.21.15.37:0/3760804418 wait complete. 2024-03-29T11:37:37.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:37.299+0000 7f192d2e0640 1 librados: shutdown 2024-03-29T11:37:37.318 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:44.125 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:44.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.272+0000 7fc3e1ec0640 10 monclient: get_monmap_and_config 2024-03-29T11:37:44.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.272+0000 7fc3e1ec0640 10 monclient: build_initial_monmap 2024-03-29T11:37:44.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.272+0000 7fc3e1ec0640 10 monclient: monmap: 2024-03-29T11:37:44.274 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:44.274 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:44.274 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:37:44.274305+0000 2024-03-29T11:37:44.274 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:37:44.274305+0000 2024-03-29T11:37:44.274 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:44.274 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:44.274 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:44.274 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:37:44.274 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:37:44.274 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:44.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.273+0000 7fc3e1ec0640 1 Processor -- start 2024-03-29T11:37:44.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.275+0000 7fc3e1ec0640 1 -- start start 2024-03-29T11:37:44.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.275+0000 7fc3e1ec0640 10 monclient: init 2024-03-29T11:37:44.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.275+0000 7fc3e1ec0640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:44.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.275+0000 7fc3e1ec0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:44.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.275+0000 7fc3e1ec0640 10 monclient(hunting): picked mon.noname-c con 0x7fc3dc0b0770 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:44.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.275+0000 7fc3e1ec0640 10 monclient(hunting): picked mon.noname-a con 0x7fc3dc14a700 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:44.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.275+0000 7fc3e1ec0640 10 monclient(hunting): picked mon.noname-b con 0x7fc3dc0ac5a0 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:44.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.275+0000 7fc3e1ec0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3dc0aba70 con 0x7fc3dc14a700 2024-03-29T11:37:44.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.276+0000 7fc3e1ec0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3dc0aecb0 con 0x7fc3dc0b0770 2024-03-29T11:37:44.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.276+0000 7fc3e1ec0640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3dc0af150 con 0x7fc3dc0ac5a0 2024-03-29T11:37:44.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.276+0000 7fc3e1ec0640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:44.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.276+0000 7fc3e1ec0640 10 monclient(hunting): authenticate will time out at 1043.725951s 2024-03-29T11:37:44.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.282+0000 7fc3daffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc3dc14a700 0x7fc3dc0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36995/0 (socket says 172.21.15.37:36995) 2024-03-29T11:37:44.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.282+0000 7fc3daffd640 1 -- 172.21.15.37:0/1017640076 learned_addr learned my addr 172.21.15.37:0/1017640076 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:44.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.282+0000 7fc3da7fc640 1 -- 172.21.15.37:0/1017640076 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535775050 0 0) 0x7fc3dc0aba70 con 0x7fc3dc14a700 2024-03-29T11:37:44.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.282+0000 7fc3da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:44.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.282+0000 7fc3da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:44.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.282+0000 7fc3da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:44.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.282+0000 7fc3da7fc640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:37:44.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.282+0000 7fc3da7fc640 1 -- 172.21.15.37:0/1017640076 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c4003520 con 0x7fc3dc14a700 2024-03-29T11:37:44.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.283+0000 7fc3da7fc640 1 -- 172.21.15.37:0/1017640076 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696829124 0 0) 0x7fc3dc0af150 con 0x7fc3dc0ac5a0 2024-03-29T11:37:44.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.283+0000 7fc3da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:44.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.283+0000 7fc3da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:44.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.283+0000 7fc3da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:44.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.283+0000 7fc3da7fc640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:37:44.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.283+0000 7fc3da7fc640 1 -- 172.21.15.37:0/1017640076 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3dc0aba70 con 0x7fc3dc0ac5a0 2024-03-29T11:37:44.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.283+0000 7fc3da7fc640 1 -- 172.21.15.37:0/1017640076 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4257517405 0 0) 0x7fc3c4003520 con 0x7fc3dc14a700 2024-03-29T11:37:44.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.283+0000 7fc3da7fc640 1 -- 172.21.15.37:0/1017640076 >> v1:172.21.15.37:6790/0 conn(0x7fc3dc0ac5a0 legacy=0x7fc3dc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 1 -- 172.21.15.37:0/1017640076 >> v1:172.21.15.146:6789/0 conn(0x7fc3dc0b0770 legacy=0x7fc3dc149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:44.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 1 monclient: found mon.noname-a 2024-03-29T11:37:44.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:44.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:37:44.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 1 -- 172.21.15.37:0/1017640076 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc0ac9d0 con 0x7fc3dc14a700 2024-03-29T11:37:44.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:44.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 1 -- 172.21.15.37:0/1017640076 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc3d0001690 con 0x7fc3dc14a700 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 10 monclient: dump: 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:44.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3e1ec0640 5 monclient: authenticate success, global_id 4317 2024-03-29T11:37:44.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3e1ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:44.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 1 -- 172.21.15.37:0/1017640076 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3d00017f0 con 0x7fc3dc14a700 2024-03-29T11:37:44.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:44.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3da7fc640 1 -- 172.21.15.37:0/1017640076 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc3d0001c40 con 0x7fc3dc14a700 2024-03-29T11:37:44.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3e1ec0640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:44.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.284+0000 7fc3e1ec0640 4 set_mon_vals no callback set 2024-03-29T11:37:44.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.285+0000 7fc3e1ec0640 10 monclient: shutdown 2024-03-29T11:37:44.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.285+0000 7fc3e1ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:44.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.285+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1017640076 >> v1:172.21.15.37:6789/0 conn(0x7fc3dc14a700 legacy=0x7fc3dc0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.285+0000 7fc3da7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:37:44.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.290+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1017640076 shutdown_connections 2024-03-29T11:37:44.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.290+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1017640076 >> 172.21.15.37:0/1017640076 conn(0x7fc3dc06b660 msgr2=0x7fc3dc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:44.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.291+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1017640076 shutdown_connections 2024-03-29T11:37:44.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.293+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1017640076 wait complete. 2024-03-29T11:37:44.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.294+0000 7fc3e1ec0640 10 monclient: build_initial_monmap 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.294+0000 7fc3e1ec0640 10 monclient: monmap: 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.295+0000 7fc3e1ec0640 1 librados: starting msgr at 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.295+0000 7fc3e1ec0640 1 librados: starting objecter 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.295+0000 7fc3e1ec0640 1 Processor -- start 2024-03-29T11:37:44.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.296+0000 7fc3e1ec0640 1 -- start start 2024-03-29T11:37:44.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.296+0000 7fc3e1ec0640 1 librados: setting wanted keys 2024-03-29T11:37:44.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.296+0000 7fc3e1ec0640 1 librados: calling monclient init 2024-03-29T11:37:44.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.296+0000 7fc3e1ec0640 10 monclient: init 2024-03-29T11:37:44.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.300+0000 7fc3e1ec0640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:44.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.300+0000 7fc3e1ec0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:44.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.300+0000 7fc3e1ec0640 10 monclient(hunting): picked mon.noname-b con 0x7fc3dc0ac5a0 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:44.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.300+0000 7fc3e1ec0640 10 monclient(hunting): picked mon.noname-a con 0x7fc3dc0b0770 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:44.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.300+0000 7fc3e1ec0640 10 monclient(hunting): picked mon.noname-c con 0x7fc3dc14a700 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:44.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.300+0000 7fc3e1ec0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3dc122620 con 0x7fc3dc0b0770 2024-03-29T11:37:44.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.300+0000 7fc3e1ec0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3dc122900 con 0x7fc3dc0ac5a0 2024-03-29T11:37:44.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.300+0000 7fc3e1ec0640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3dc120f50 con 0x7fc3dc14a700 2024-03-29T11:37:44.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.300+0000 7fc3e1ec0640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:44.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.300+0000 7fc3e1ec0640 10 monclient(hunting): authenticate will time out at 1043.750626s 2024-03-29T11:37:44.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.304+0000 7fc3dbfff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fc3dc14a700 0x7fc3dc1ef8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:51079/0 (socket says 172.21.15.37:51079) 2024-03-29T11:37:44.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.304+0000 7fc3dbfff640 1 -- 172.21.15.37:0/1988749375 learned_addr learned my addr 172.21.15.37:0/1988749375 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:44.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.304+0000 7fc3c3fff640 10 client.?.objecter ms_handle_connect 0x7fc3dc14a700 2024-03-29T11:37:44.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.305+0000 7fc3c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:44.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.305+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433474353 0 0) 0x7fc3dc120f50 con 0x7fc3dc14a700 2024-03-29T11:37:44.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.305+0000 7fc3c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:44.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.305+0000 7fc3c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:44.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.305+0000 7fc3c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:44.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.305+0000 7fc3c3fff640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:37:44.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.305+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c8004eb0 con 0x7fc3dc14a700 2024-03-29T11:37:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.305+0000 7fc3c3fff640 10 client.?.objecter ms_handle_connect 0x7fc3dc0b0770 2024-03-29T11:37:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.305+0000 7fc3c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.305+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874635667 0 0) 0x7fc3dc122620 con 0x7fc3dc0b0770 2024-03-29T11:37:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.305+0000 7fc3c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.305+0000 7fc3c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.305+0000 7fc3c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.305+0000 7fc3c3fff640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:37:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3dc120f50 con 0x7fc3dc0b0770 2024-03-29T11:37:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1015247902 0 0) 0x7fc3c8004eb0 con 0x7fc3dc14a700 2024-03-29T11:37:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3dc122620 con 0x7fc3dc14a700 2024-03-29T11:37:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc3cc001a50 con 0x7fc3dc14a700 2024-03-29T11:37:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:44.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:44.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 10 monclient(hunting): dump: 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2013143129 0 0) 0x7fc3dc122620 con 0x7fc3dc14a700 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 >> v1:172.21.15.146:6789/0 conn(0x7fc3dc0ac5a0 legacy=0x7fc3dc1e3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 >> v1:172.21.15.37:6789/0 conn(0x7fc3dc0b0770 legacy=0x7fc3dc147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:44.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 1 monclient: found mon.c 2024-03-29T11:37:44.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:44.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:44.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc1e46b0 con 0x7fc3dc14a700 2024-03-29T11:37:44.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:37:44.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:44.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:44.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 10 client.?.objecter ms_handle_connect 0x7fc3dc0ac5a0 2024-03-29T11:37:44.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:44.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3cc0035f0 con 0x7fc3dc14a700 2024-03-29T11:37:44.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:44.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.306+0000 7fc3d9ffb640 4 set_mon_vals no callback set 2024-03-29T11:37:44.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.307+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc3cc003a00 con 0x7fc3dc14a700 2024-03-29T11:37:44.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.307+0000 7fc3c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.307+0000 7fc3c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.307+0000 7fc3c3fff640 10 monclient: dump: 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.310+0000 7fc3e1ec0640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.310+0000 7fc3e1ec0640 10 monclient: _renew_subs 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.310+0000 7fc3e1ec0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:44.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.310+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1988749375 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3dc121470 con 0x7fc3dc14a700 2024-03-29T11:37:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.310+0000 7fc3e1ec0640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.310+0000 7fc3e1ec0640 10 monclient: _renew_subs 2024-03-29T11:37:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.310+0000 7fc3e1ec0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.310+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1988749375 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3dc123250 con 0x7fc3dc14a700 2024-03-29T11:37:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.310+0000 7fc3e1ec0640 1 librados: init done 2024-03-29T11:37:44.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.312+0000 7fc3e1ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:44.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.312+0000 7fc3e1ec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:44.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.312+0000 7fc3e1ec0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:44.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.312+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1988749375 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3dc0ab540 con 0x7fc3dc14a700 2024-03-29T11:37:44.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.314+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fc3cc003b60 con 0x7fc3dc14a700 2024-03-29T11:37:44.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.315+0000 7fc3db7fe640 10 monclient: get_auth_request con 0x7fc3c806dc90 auth_method 0 2024-03-29T11:37:44.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.315+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fc3cc084ab0 con 0x7fc3dc14a700 2024-03-29T11:37:44.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.315+0000 7fc3c3fff640 10 client.4337.objecter ms_dispatch 0x7fc3dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:44.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.315+0000 7fc3c3fff640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:44.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.315+0000 7fc3c3fff640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:44.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.315+0000 7fc3c3fff640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T11:37:44.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.315+0000 7fc3c3fff640 10 client.4337.objecter ms_handle_connect 0x7fc3c806dc90 2024-03-29T11:37:44.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.327+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3cc050390 con 0x7fc3dc14a700 2024-03-29T11:37:44.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.327+0000 7fc3c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:44.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.327+0000 7fc3c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:44.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.665+0000 7fc3e1ec0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:44.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.665+0000 7fc3e1ec0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:44.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.665+0000 7fc3e1ec0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:44.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.665+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1988749375 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3dc1210e0 con 0x7fc3dc14a700 2024-03-29T11:37:44.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.667+0000 7fc3c3fff640 1 -- 172.21.15.37:0/1988749375 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fc3cc0522c0 con 0x7fc3dc14a700 2024-03-29T11:37:44.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.667+0000 7fc3c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:44.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.667+0000 7fc3c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:44.669 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:37:44.669 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:44.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.670+0000 7fc3e1ec0640 10 librados: watch_flush enter 2024-03-29T11:37:44.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.671+0000 7fc3e1ec0640 10 librados: watch_flush exit 2024-03-29T11:37:44.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.671+0000 7fc3e1ec0640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T11:37:44.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.671+0000 7fc3e1ec0640 10 client.4337.objecter successfully canceled tick 2024-03-29T11:37:44.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.671+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1988749375 >> v1:172.21.15.146:6816/2899149139 conn(0x7fc3c806dc90 legacy=0x7fc3c8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.671+0000 7fc3e1ec0640 10 monclient: shutdown 2024-03-29T11:37:44.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.671+0000 7fc3e1ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:44.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.671+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1988749375 >> v1:172.21.15.37:6790/0 conn(0x7fc3dc14a700 legacy=0x7fc3dc1ef8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.673+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1988749375 shutdown_connections 2024-03-29T11:37:44.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.673+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1988749375 >> 172.21.15.37:0/1988749375 conn(0x7fc3dc06b660 msgr2=0x7fc3dc0b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:44.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.673+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1988749375 shutdown_connections 2024-03-29T11:37:44.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.673+0000 7fc3e1ec0640 1 -- 172.21.15.37:0/1988749375 wait complete. 2024-03-29T11:37:44.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:44.673+0000 7fc3e1ec0640 1 librados: shutdown 2024-03-29T11:37:44.691 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:51.498 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:51.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.650+0000 7f2eae6c8640 10 monclient: get_monmap_and_config 2024-03-29T11:37:51.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.650+0000 7f2eae6c8640 10 monclient: build_initial_monmap 2024-03-29T11:37:51.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.650+0000 7f2eae6c8640 10 monclient: monmap: 2024-03-29T11:37:51.651 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:51.651 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:51.651 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:37:51.651384+0000 2024-03-29T11:37:51.651 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:37:51.651384+0000 2024-03-29T11:37:51.651 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:51.651 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:51.651 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:51.652 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:37:51.652 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:37:51.652 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:51.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.653+0000 7f2eae6c8640 1 Processor -- start 2024-03-29T11:37:51.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.653+0000 7f2eae6c8640 1 -- start start 2024-03-29T11:37:51.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.653+0000 7f2eae6c8640 10 monclient: init 2024-03-29T11:37:51.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.654+0000 7f2eae6c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:51.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.654+0000 7f2eae6c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:51.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.654+0000 7f2eae6c8640 10 monclient(hunting): picked mon.noname-c con 0x7f2ea80b0860 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:51.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.654+0000 7f2eae6c8640 10 monclient(hunting): picked mon.noname-b con 0x7f2ea813ef30 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:51.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.654+0000 7f2eae6c8640 10 monclient(hunting): picked mon.noname-a con 0x7f2ea80af8e0 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:51.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.654+0000 7f2eae6c8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea80b0c40 con 0x7f2ea80af8e0 2024-03-29T11:37:51.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.654+0000 7f2eae6c8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea80aed10 con 0x7f2ea80b0860 2024-03-29T11:37:51.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.654+0000 7f2eae6c8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea80afd50 con 0x7f2ea813ef30 2024-03-29T11:37:51.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.654+0000 7f2eae6c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:51.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.654+0000 7f2eae6c8640 10 monclient(hunting): authenticate will time out at 1051.103119s 2024-03-29T11:37:51.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.657+0000 7f2eacc65640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f2ea80af8e0 0x7f2ea8149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55889/0 (socket says 172.21.15.37:55889) 2024-03-29T11:37:51.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.657+0000 7f2eacc65640 1 -- 172.21.15.37:0/3558898542 learned_addr learned my addr 172.21.15.37:0/3558898542 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:51.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 1 -- 172.21.15.37:0/3558898542 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528991216 0 0) 0x7f2ea80aed10 con 0x7f2ea80b0860 2024-03-29T11:37:51.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:51.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:51.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:51.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:37:51.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 1 -- 172.21.15.37:0/3558898542 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e90003520 con 0x7f2ea80b0860 2024-03-29T11:37:51.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 1 -- 172.21.15.37:0/3558898542 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1992964224 0 0) 0x7f2e90003520 con 0x7f2ea80b0860 2024-03-29T11:37:51.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 1 -- 172.21.15.37:0/3558898542 >> v1:172.21.15.37:6790/0 conn(0x7f2ea813ef30 legacy=0x7f2ea80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:51.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 1 -- 172.21.15.37:0/3558898542 >> v1:172.21.15.37:6789/0 conn(0x7f2ea80af8e0 legacy=0x7f2ea8149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:51.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:51.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:37:51.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:51.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:37:51.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 1 -- 172.21.15.37:0/3558898542 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea80ab1b0 con 0x7f2ea80b0860 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 1 -- 172.21.15.37:0/3558898542 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2e9c001690 con 0x7f2ea80b0860 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient: dump: 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:51.663 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 1 -- 172.21.15.37:0/3558898542 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e9c001860 con 0x7f2ea80b0860 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 1 -- 172.21.15.37:0/3558898542 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2e9c001cb0 con 0x7f2ea80b0860 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.658+0000 7f2ea6ffd640 10 monclient: dump: 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:51.664 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:51.665 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:51.665 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:51.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.659+0000 7f2eae6c8640 5 monclient: authenticate success, global_id 4321 2024-03-29T11:37:51.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.659+0000 7f2eae6c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:51.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.659+0000 7f2eae6c8640 4 set_mon_vals no callback set 2024-03-29T11:37:51.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.660+0000 7f2eae6c8640 10 monclient: shutdown 2024-03-29T11:37:51.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.660+0000 7f2eae6c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:51.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.660+0000 7f2eae6c8640 1 -- 172.21.15.37:0/3558898542 >> v1:172.21.15.146:6789/0 conn(0x7f2ea80b0860 legacy=0x7f2ea813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:51.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.660+0000 7f2eae6c8640 1 -- 172.21.15.37:0/3558898542 shutdown_connections 2024-03-29T11:37:51.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.660+0000 7f2eae6c8640 1 -- 172.21.15.37:0/3558898542 >> 172.21.15.37:0/3558898542 conn(0x7f2ea806b660 msgr2=0x7f2ea806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:51.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.662+0000 7f2eae6c8640 1 -- 172.21.15.37:0/3558898542 shutdown_connections 2024-03-29T11:37:51.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.662+0000 7f2eae6c8640 1 -- 172.21.15.37:0/3558898542 wait complete. 2024-03-29T11:37:51.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.663+0000 7f2eae6c8640 10 monclient: build_initial_monmap 2024-03-29T11:37:51.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.663+0000 7f2eae6c8640 10 monclient: monmap: 2024-03-29T11:37:51.666 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:51.666 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.663+0000 7f2eae6c8640 1 librados: starting msgr at 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.663+0000 7f2eae6c8640 1 librados: starting objecter 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.664+0000 7f2eae6c8640 1 Processor -- start 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.666+0000 7f2eae6c8640 1 -- start start 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.666+0000 7f2eae6c8640 1 librados: setting wanted keys 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.666+0000 7f2eae6c8640 1 librados: calling monclient init 2024-03-29T11:37:51.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.666+0000 7f2eae6c8640 10 monclient: init 2024-03-29T11:37:51.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.667+0000 7f2eae6c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:51.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.667+0000 7f2eae6c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:51.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.667+0000 7f2eae6c8640 10 monclient(hunting): picked mon.noname-c con 0x7f2ea80af8e0 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:51.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.667+0000 7f2eae6c8640 10 monclient(hunting): picked mon.noname-b con 0x7f2ea80b0860 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:51.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.667+0000 7f2eae6c8640 10 monclient(hunting): picked mon.noname-a con 0x7f2ea813ef30 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:51.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.667+0000 7f2eae6c8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea8135c70 con 0x7f2ea813ef30 2024-03-29T11:37:51.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.667+0000 7f2eae6c8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea80b2620 con 0x7f2ea80b0860 2024-03-29T11:37:51.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.667+0000 7f2eae6c8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea80b35e0 con 0x7f2ea80af8e0 2024-03-29T11:37:51.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.667+0000 7f2eae6c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:51.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.667+0000 7f2eae6c8640 10 monclient(hunting): authenticate will time out at 1051.116442s 2024-03-29T11:37:51.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.673+0000 7f2ea7fff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f2ea80af8e0 0x7f2ea81d62b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:44175/0 (socket says 172.21.15.37:44175) 2024-03-29T11:37:51.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.673+0000 7f2ea7fff640 1 -- 172.21.15.37:0/902115211 learned_addr learned my addr 172.21.15.37:0/902115211 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:51.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.674+0000 7f2ea4ff9640 10 client.?.objecter ms_handle_connect 0x7f2ea80af8e0 2024-03-29T11:37:51.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.674+0000 7f2ea4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:51.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.674+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692906222 0 0) 0x7f2ea80b35e0 con 0x7f2ea80af8e0 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.674+0000 7f2ea4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.674+0000 7f2ea4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.674+0000 7f2ea4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.674+0000 7f2ea4ff9640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.674+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e94004eb0 con 0x7f2ea80af8e0 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.674+0000 7f2ea4ff9640 10 client.?.objecter ms_handle_connect 0x7f2ea80b0860 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.674+0000 7f2ea4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.675+0000 7f2ea4ff9640 10 client.?.objecter ms_handle_connect 0x7f2ea813ef30 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.675+0000 7f2ea4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.675+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151099811 0 0) 0x7f2ea80b2620 con 0x7f2ea80b0860 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.675+0000 7f2ea4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.675+0000 7f2ea4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.675+0000 7f2ea4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.675+0000 7f2ea4ff9640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:37:51.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.675+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea80b35e0 con 0x7f2ea80b0860 2024-03-29T11:37:51.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.675+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291486494 0 0) 0x7f2ea8135c70 con 0x7f2ea813ef30 2024-03-29T11:37:51.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.675+0000 7f2ea4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:51.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.675+0000 7f2ea4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:51.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.675+0000 7f2ea4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:51.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.675+0000 7f2ea4ff9640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:37:51.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.675+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea80b2620 con 0x7f2ea813ef30 2024-03-29T11:37:51.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2419166857 0 0) 0x7f2ea80b35e0 con 0x7f2ea80b0860 2024-03-29T11:37:51.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ea8135c70 con 0x7f2ea80b0860 2024-03-29T11:37:51.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2ea0001af0 con 0x7f2ea80b0860 2024-03-29T11:37:51.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:51.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:51.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:51.682 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:51.682 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:51.683 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:51.683 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:51.683 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:51.683 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:51.684 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:51.684 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:51.684 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:51.684 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:51.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:51.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3224356654 0 0) 0x7f2ea80b2620 con 0x7f2ea813ef30 2024-03-29T11:37:51.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ea80b35e0 con 0x7f2ea813ef30 2024-03-29T11:37:51.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2e98001550 con 0x7f2ea813ef30 2024-03-29T11:37:51.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:51.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:51.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:51.684 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1711333854 0 0) 0x7f2ea8135c70 con 0x7f2ea80b0860 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.676+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 >> v1:172.21.15.37:6790/0 conn(0x7f2ea80af8e0 legacy=0x7f2ea81d62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 >> v1:172.21.15.37:6789/0 conn(0x7f2ea813ef30 legacy=0x7f2ea81dd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 1 monclient: found mon.b 2024-03-29T11:37:51.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:51.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:37:51.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea80b3a80 con 0x7f2ea80b0860 2024-03-29T11:37:51.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:37:51.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:37:51.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:51.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2eae6c8640 5 monclient: authenticate success, global_id 4324 2024-03-29T11:37:51.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2eae6c8640 10 monclient: _renew_subs 2024-03-29T11:37:51.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2eae6c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:37:51.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2eae6c8640 1 -- 172.21.15.37:0/902115211 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea80b3150 con 0x7f2ea80b0860 2024-03-29T11:37:51.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2eae6c8640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:51.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2eae6c8640 10 monclient: _renew_subs 2024-03-29T11:37:51.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2eae6c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:37:51.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2eae6c8640 1 -- 172.21.15.37:0/902115211 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea813ae10 con 0x7f2ea80b0860 2024-03-29T11:37:51.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2eae6c8640 1 librados: init done 2024-03-29T11:37:51.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ea0001fe0 con 0x7f2ea80b0860 2024-03-29T11:37:51.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:51.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea67fc640 4 set_mon_vals no callback set 2024-03-29T11:37:51.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2ea0003b10 con 0x7f2ea80b0860 2024-03-29T11:37:51.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:51.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:51.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.677+0000 7f2ea4ff9640 10 monclient: dump: 2024-03-29T11:37:51.687 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:51.687 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:51.687 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:51.688 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:51.688 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:51.688 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:51.688 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:51.688 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:51.688 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:51.688 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:51.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.679+0000 7f2eae6c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:51.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.681+0000 7f2eae6c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:51.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.681+0000 7f2eae6c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:37:51.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.681+0000 7f2eae6c8640 1 -- 172.21.15.37:0/902115211 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e88005080 con 0x7f2ea80b0860 2024-03-29T11:37:51.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.682+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f2ea0019ae0 con 0x7f2ea80b0860 2024-03-29T11:37:51.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.683+0000 7f2ea7fff640 10 monclient: get_auth_request con 0x7f2e9406dcd0 auth_method 0 2024-03-29T11:37:51.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.683+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f2ea00839a0 con 0x7f2ea80b0860 2024-03-29T11:37:51.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.683+0000 7f2ea4ff9640 10 client.4324.objecter ms_dispatch 0x7f2ea8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:51.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.683+0000 7f2ea4ff9640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:51.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.683+0000 7f2ea4ff9640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:51.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.683+0000 7f2ea4ff9640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-29T11:37:51.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.683+0000 7f2ea4ff9640 10 client.4324.objecter ms_handle_connect 0x7f2e9406dcd0 2024-03-29T11:37:51.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.690+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ea004f280 con 0x7f2ea80b0860 2024-03-29T11:37:51.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.690+0000 7f2ea4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:51.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:51.690+0000 7f2ea4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:52.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.014+0000 7f2eae6c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:52.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.014+0000 7f2eae6c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:52.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.014+0000 7f2eae6c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:37:52.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.014+0000 7f2eae6c8640 1 -- 172.21.15.37:0/902115211 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e88005330 con 0x7f2ea80b0860 2024-03-29T11:37:52.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.017+0000 7f2ea4ff9640 1 -- 172.21.15.37:0/902115211 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f2ea00511b0 con 0x7f2ea80b0860 2024-03-29T11:37:52.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.017+0000 7f2ea4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:52.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.017+0000 7f2ea4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:52.017 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:37:52.017 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:52.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.020+0000 7f2eae6c8640 10 librados: watch_flush enter 2024-03-29T11:37:52.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.020+0000 7f2eae6c8640 10 librados: watch_flush exit 2024-03-29T11:37:52.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.020+0000 7f2eae6c8640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-29T11:37:52.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.020+0000 7f2eae6c8640 10 client.4324.objecter successfully canceled tick 2024-03-29T11:37:52.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.021+0000 7f2eae6c8640 1 -- 172.21.15.37:0/902115211 >> v1:172.21.15.146:6816/2899149139 conn(0x7f2e9406dcd0 legacy=0x7f2e94070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:52.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.021+0000 7f2eae6c8640 10 monclient: shutdown 2024-03-29T11:37:52.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.021+0000 7f2eae6c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:52.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.021+0000 7f2eae6c8640 1 -- 172.21.15.37:0/902115211 >> v1:172.21.15.146:6789/0 conn(0x7f2ea80b0860 legacy=0x7f2ea81d69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:52.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.021+0000 7f2eae6c8640 1 -- 172.21.15.37:0/902115211 shutdown_connections 2024-03-29T11:37:52.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.022+0000 7f2eae6c8640 1 -- 172.21.15.37:0/902115211 >> 172.21.15.37:0/902115211 conn(0x7f2ea806b660 msgr2=0x7f2ea81cd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:52.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.022+0000 7f2eae6c8640 1 -- 172.21.15.37:0/902115211 shutdown_connections 2024-03-29T11:37:52.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.025+0000 7f2eae6c8640 1 -- 172.21.15.37:0/902115211 wait complete. 2024-03-29T11:37:52.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:52.025+0000 7f2eae6c8640 1 librados: shutdown 2024-03-29T11:37:52.043 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:58.849 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:58.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:58.997+0000 7fd3f2f2d640 10 monclient: get_monmap_and_config 2024-03-29T11:37:58.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:58.997+0000 7fd3f2f2d640 10 monclient: build_initial_monmap 2024-03-29T11:37:58.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:58.997+0000 7fd3f2f2d640 10 monclient: monmap: 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:37:58.998591+0000 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:37:58.998591+0000 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:58.998+0000 7fd3f2f2d640 1 Processor -- start 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:58.998+0000 7fd3f2f2d640 1 -- start start 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:58.998+0000 7fd3f2f2d640 10 monclient: init 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:58.999+0000 7fd3f2f2d640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:59.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:58.999+0000 7fd3f2f2d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:59.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:58.999+0000 7fd3f2f2d640 10 monclient(hunting): picked mon.noname-c con 0x7fd3ec0a9750 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:59.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:58.999+0000 7fd3f2f2d640 10 monclient(hunting): picked mon.noname-b con 0x7fd3ec0adca0 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:59.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.000+0000 7fd3f2f2d640 10 monclient(hunting): picked mon.noname-a con 0x7fd3ec146a80 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:59.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.000+0000 7fd3f2f2d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ec0abdc0 con 0x7fd3ec146a80 2024-03-29T11:37:59.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.000+0000 7fd3f2f2d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ec0b0070 con 0x7fd3ec0a9750 2024-03-29T11:37:59.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.000+0000 7fd3f2f2d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ec0af880 con 0x7fd3ec0adca0 2024-03-29T11:37:59.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.000+0000 7fd3f2f2d640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:59.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.000+0000 7fd3f2f2d640 10 monclient(hunting): authenticate will time out at 1058.449558s 2024-03-29T11:37:59.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.003+0000 7fd3f272c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd3ec146a80 0x7fd3ec151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43461/0 (socket says 172.21.15.37:43461) 2024-03-29T11:37:59.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.003+0000 7fd3f272c640 1 -- 172.21.15.37:0/1110309477 learned_addr learned my addr 172.21.15.37:0/1110309477 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:59.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.004+0000 7fd3f0f29640 1 -- 172.21.15.37:0/1110309477 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898083998 0 0) 0x7fd3ec0abdc0 con 0x7fd3ec146a80 2024-03-29T11:37:59.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.004+0000 7fd3f0f29640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.004+0000 7fd3f0f29640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.004+0000 7fd3f0f29640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.004+0000 7fd3f0f29640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:37:59.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.004+0000 7fd3f0f29640 1 -- 172.21.15.37:0/1110309477 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d8003520 con 0x7fd3ec146a80 2024-03-29T11:37:59.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.004+0000 7fd3f0f29640 1 -- 172.21.15.37:0/1110309477 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 941111829 0 0) 0x7fd3d8003520 con 0x7fd3ec146a80 2024-03-29T11:37:59.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.004+0000 7fd3f0f29640 1 -- 172.21.15.37:0/1110309477 >> v1:172.21.15.37:6790/0 conn(0x7fd3ec0adca0 legacy=0x7fd3ec146370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:59.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 1 -- 172.21.15.37:0/1110309477 >> v1:172.21.15.146:6789/0 conn(0x7fd3ec0a9750 legacy=0x7fd3ec143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:59.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 10 monclient: _finish_hunting 0 2024-03-29T11:37:59.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 1 monclient: found mon.noname-a 2024-03-29T11:37:59.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:59.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 1 -- 172.21.15.37:0/1110309477 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec0ac030 con 0x7fd3ec146a80 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 10 monclient: _finish_auth 0 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 10 monclient: _check_auth_tickets 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 1 -- 172.21.15.37:0/1110309477 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fd3e4001a10 con 0x7fd3ec146a80 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 10 monclient: dump: 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:59.009 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:59.010 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:59.010 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:59.010 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:59.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f2f2d640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:37:59.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f2f2d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:59.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 1 -- 172.21.15.37:0/1110309477 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3e4001d20 con 0x7fd3ec146a80 2024-03-29T11:37:59.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f0f29640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:59.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f2f2d640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:59.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.005+0000 7fd3f2f2d640 4 set_mon_vals no callback set 2024-03-29T11:37:59.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.006+0000 7fd3f2f2d640 10 monclient: shutdown 2024-03-29T11:37:59.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.006+0000 7fd3f2f2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:59.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.006+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/1110309477 >> v1:172.21.15.37:6789/0 conn(0x7fd3ec146a80 legacy=0x7fd3ec151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.006+0000 7fd3f0f29640 1 -- 172.21.15.37:0/1110309477 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fd3e40020b0 con 0x7fd3ec146a80 2024-03-29T11:37:59.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.007+0000 7fd3f0f29640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:37:59.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.008+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/1110309477 shutdown_connections 2024-03-29T11:37:59.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.008+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/1110309477 >> 172.21.15.37:0/1110309477 conn(0x7fd3ec0673e0 msgr2=0x7fd3ec0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:59.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.010+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/1110309477 shutdown_connections 2024-03-29T11:37:59.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.010+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/1110309477 wait complete. 2024-03-29T11:37:59.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.011+0000 7fd3f2f2d640 10 monclient: build_initial_monmap 2024-03-29T11:37:59.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.011+0000 7fd3f2f2d640 10 monclient: monmap: 2024-03-29T11:37:59.012 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:37:59.012 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:59.012 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:37:59.013 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:37:59.013 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:59.013 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:59.013 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:37:59.013 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:37:59.013 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:37:59.013 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:59.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.012+0000 7fd3f2f2d640 1 librados: starting msgr at 2024-03-29T11:37:59.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.012+0000 7fd3f2f2d640 1 librados: starting objecter 2024-03-29T11:37:59.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.012+0000 7fd3f2f2d640 1 Processor -- start 2024-03-29T11:37:59.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.012+0000 7fd3f2f2d640 1 -- start start 2024-03-29T11:37:59.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.012+0000 7fd3f2f2d640 1 librados: setting wanted keys 2024-03-29T11:37:59.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.012+0000 7fd3f2f2d640 1 librados: calling monclient init 2024-03-29T11:37:59.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.012+0000 7fd3f2f2d640 10 monclient: init 2024-03-29T11:37:59.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.013+0000 7fd3f2f2d640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:59.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.013+0000 7fd3f2f2d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:59.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.013+0000 7fd3f2f2d640 10 monclient(hunting): picked mon.noname-c con 0x7fd3ec0a9750 addr v1:172.21.15.37:6790/0 2024-03-29T11:37:59.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.013+0000 7fd3f2f2d640 10 monclient(hunting): picked mon.noname-b con 0x7fd3ec0adca0 addr v1:172.21.15.146:6789/0 2024-03-29T11:37:59.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.013+0000 7fd3f2f2d640 10 monclient(hunting): picked mon.noname-a con 0x7fd3ec146a80 addr v1:172.21.15.37:6789/0 2024-03-29T11:37:59.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.013+0000 7fd3f2f2d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ec1cf700 con 0x7fd3ec146a80 2024-03-29T11:37:59.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.013+0000 7fd3f2f2d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ec06f4d0 con 0x7fd3ec0adca0 2024-03-29T11:37:59.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.013+0000 7fd3f2f2d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ec06f7b0 con 0x7fd3ec0a9750 2024-03-29T11:37:59.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.013+0000 7fd3f2f2d640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:59.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.014+0000 7fd3f2f2d640 10 monclient(hunting): authenticate will time out at 1058.462988s 2024-03-29T11:37:59.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.017+0000 7fd3f272c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd3ec146a80 0x7fd3ec1d02a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43683/0 (socket says 172.21.15.37:43683) 2024-03-29T11:37:59.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.017+0000 7fd3f272c640 1 -- 172.21.15.37:0/3019722944 learned_addr learned my addr 172.21.15.37:0/3019722944 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:37:59.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.017+0000 7fd3e27fc640 10 client.?.objecter ms_handle_connect 0x7fd3ec0a9750 2024-03-29T11:37:59.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.017+0000 7fd3e27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:59.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.017+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400936542 0 0) 0x7fd3ec06f7b0 con 0x7fd3ec0a9750 2024-03-29T11:37:59.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.017+0000 7fd3e27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.017+0000 7fd3e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.017+0000 7fd3e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.017+0000 7fd3e27fc640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:37:59.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.018+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d0004eb0 con 0x7fd3ec0a9750 2024-03-29T11:37:59.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1375212230 0 0) 0x7fd3d0004eb0 con 0x7fd3ec0a9750 2024-03-29T11:37:59.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3d0005300 con 0x7fd3ec0a9750 2024-03-29T11:37:59.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fd3e8001830 con 0x7fd3ec0a9750 2024-03-29T11:37:59.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 10 monclient(hunting): dump: 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1006364180 0 0) 0x7fd3d0005300 con 0x7fd3ec0a9750 2024-03-29T11:37:59.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 >> v1:172.21.15.146:6789/0 conn(0x7fd3ec0adca0 legacy=0x7fd3ec1cfb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 >> v1:172.21.15.37:6789/0 conn(0x7fd3ec146a80 legacy=0x7fd3ec1d02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 1 monclient: found mon.c 2024-03-29T11:37:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec1234f0 con 0x7fd3ec0a9750 2024-03-29T11:37:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3f2f2d640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:37:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.019+0000 7fd3f2f2d640 10 monclient: _renew_subs 2024-03-29T11:37:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.020+0000 7fd3f2f2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:59.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.020+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/3019722944 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ec1ef440 con 0x7fd3ec0a9750 2024-03-29T11:37:59.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.020+0000 7fd3f2f2d640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:59.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.020+0000 7fd3f2f2d640 10 monclient: _renew_subs 2024-03-29T11:37:59.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.020+0000 7fd3f2f2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:59.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.020+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/3019722944 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ec1ef7b0 con 0x7fd3ec0a9750 2024-03-29T11:37:59.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.020+0000 7fd3f2f2d640 1 librados: init done 2024-03-29T11:37:59.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.022+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3e80035f0 con 0x7fd3ec0a9750 2024-03-29T11:37:59.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.022+0000 7fd3e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:59.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.022+0000 7fd3e3fff640 4 set_mon_vals no callback set 2024-03-29T11:37:59.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.023+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fd3e8001830 con 0x7fd3ec0a9750 2024-03-29T11:37:59.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.023+0000 7fd3e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.023+0000 7fd3e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:59.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.023+0000 7fd3e27fc640 10 monclient: dump: 2024-03-29T11:37:59.026 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:37:59.027 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:37:59.027 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:59.027 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:37:59.027 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.027 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:37:59.027 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:37:59.027 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:37:59.027 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:37:59.027 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:37:59.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.025+0000 7fd3f2f2d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:59.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.025+0000 7fd3f2f2d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:59.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.025+0000 7fd3f2f2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:59.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.025+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/3019722944 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d4005080 con 0x7fd3ec0a9750 2024-03-29T11:37:59.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.025+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fd3e8019650 con 0x7fd3ec0a9750 2024-03-29T11:37:59.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.026+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fd3e8083590 con 0x7fd3ec0a9750 2024-03-29T11:37:59.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.026+0000 7fd3e27fc640 10 client.4343.objecter ms_dispatch 0x7fd3ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:59.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.026+0000 7fd3e27fc640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:59.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.026+0000 7fd3e27fc640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:59.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.026+0000 7fd3e27fc640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-29T11:37:59.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.026+0000 7fd3f172a640 10 monclient: get_auth_request con 0x7fd3d006e020 auth_method 0 2024-03-29T11:37:59.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.027+0000 7fd3e27fc640 10 client.4343.objecter ms_handle_connect 0x7fd3d006e020 2024-03-29T11:37:59.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.035+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd3e804eca0 con 0x7fd3ec0a9750 2024-03-29T11:37:59.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.035+0000 7fd3e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:59.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.035+0000 7fd3e27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:59.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.363+0000 7fd3f2f2d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:59.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.363+0000 7fd3f2f2d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:59.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.363+0000 7fd3f2f2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:37:59.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.363+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/3019722944 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3d4005330 con 0x7fd3ec0a9750 2024-03-29T11:37:59.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.366+0000 7fd3e27fc640 1 -- 172.21.15.37:0/3019722944 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fd3e804eca0 con 0x7fd3ec0a9750 2024-03-29T11:37:59.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.366+0000 7fd3e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:59.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.366+0000 7fd3e27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:59.367 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:37:59.368 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:59.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.370+0000 7fd3f2f2d640 10 librados: watch_flush enter 2024-03-29T11:37:59.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.370+0000 7fd3f2f2d640 10 librados: watch_flush exit 2024-03-29T11:37:59.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.370+0000 7fd3f2f2d640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-29T11:37:59.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.370+0000 7fd3f2f2d640 10 client.4343.objecter successfully canceled tick 2024-03-29T11:37:59.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.371+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/3019722944 >> v1:172.21.15.146:6816/2899149139 conn(0x7fd3d006e020 legacy=0x7fd3d00704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.371+0000 7fd3f2f2d640 10 monclient: shutdown 2024-03-29T11:37:59.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.371+0000 7fd3f2f2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:59.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.371+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/3019722944 >> v1:172.21.15.37:6790/0 conn(0x7fd3ec0a9750 legacy=0x7fd3ec06db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.371+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/3019722944 shutdown_connections 2024-03-29T11:37:59.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.371+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/3019722944 >> 172.21.15.37:0/3019722944 conn(0x7fd3ec0673e0 msgr2=0x7fd3ec0aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:59.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.371+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/3019722944 shutdown_connections 2024-03-29T11:37:59.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.371+0000 7fd3f2f2d640 1 -- 172.21.15.37:0/3019722944 wait complete. 2024-03-29T11:37:59.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:37:59.371+0000 7fd3f2f2d640 1 librados: shutdown 2024-03-29T11:37:59.391 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:06.197 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:06.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.346+0000 7f69c8e96640 10 monclient: get_monmap_and_config 2024-03-29T11:38:06.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.346+0000 7f69c8e96640 10 monclient: build_initial_monmap 2024-03-29T11:38:06.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.346+0000 7f69c8e96640 10 monclient: monmap: 2024-03-29T11:38:06.347 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:06.347 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:06.347 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:38:06.347101+0000 2024-03-29T11:38:06.347 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:38:06.347101+0000 2024-03-29T11:38:06.347 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:06.347 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:06.347 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:06.347 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:38:06.347 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:38:06.347 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:06.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.349+0000 7f69c8e96640 1 Processor -- start 2024-03-29T11:38:06.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.350+0000 7f69c8e96640 1 -- start start 2024-03-29T11:38:06.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.350+0000 7f69c8e96640 10 monclient: init 2024-03-29T11:38:06.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.351+0000 7f69c8e96640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:06.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.351+0000 7f69c8e96640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:06.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.351+0000 7f69c8e96640 10 monclient(hunting): picked mon.noname-a con 0x7f69c41422b0 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:06.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.351+0000 7f69c8e96640 10 monclient(hunting): picked mon.noname-b con 0x7f69c4142690 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:06.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.351+0000 7f69c8e96640 10 monclient(hunting): picked mon.noname-c con 0x7f69c40b17d0 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:06.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.351+0000 7f69c8e96640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c4142c80 con 0x7f69c41422b0 2024-03-29T11:38:06.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.351+0000 7f69c8e96640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c40aae40 con 0x7f69c40b17d0 2024-03-29T11:38:06.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.351+0000 7f69c8e96640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c413fbd0 con 0x7f69c4142690 2024-03-29T11:38:06.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.351+0000 7f69c8e96640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:06.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.351+0000 7f69c8e96640 10 monclient(hunting): authenticate will time out at 1065.800061s 2024-03-29T11:38:06.355 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.355+0000 7f69c1d9b640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f69c4142690 0x7f69c40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:34577/0 (socket says 172.21.15.37:34577) 2024-03-29T11:38:06.355 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.355+0000 7f69c1d9b640 1 -- 172.21.15.37:0/3102317823 learned_addr learned my addr 172.21.15.37:0/3102317823 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:06.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 1 -- 172.21.15.37:0/3102317823 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867636286 0 0) 0x7f69c4142c80 con 0x7f69c41422b0 2024-03-29T11:38:06.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:06.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:06.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:06.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:38:06.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 1 -- 172.21.15.37:0/3102317823 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69ac003520 con 0x7f69c41422b0 2024-03-29T11:38:06.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 1 -- 172.21.15.37:0/3102317823 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779630491 0 0) 0x7f69c413fbd0 con 0x7f69c4142690 2024-03-29T11:38:06.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:06.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:06.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:06.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:38:06.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 1 -- 172.21.15.37:0/3102317823 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c4142c80 con 0x7f69c4142690 2024-03-29T11:38:06.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 1 -- 172.21.15.37:0/3102317823 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645586103 0 0) 0x7f69c40aae40 con 0x7f69c40b17d0 2024-03-29T11:38:06.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.357+0000 7f69c159a640 1 -- 172.21.15.37:0/3102317823 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c413fbd0 con 0x7f69c40b17d0 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 1 -- 172.21.15.37:0/3102317823 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3435245217 0 0) 0x7f69c413fbd0 con 0x7f69c40b17d0 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 1 -- 172.21.15.37:0/3102317823 >> v1:172.21.15.37:6790/0 conn(0x7f69c4142690 legacy=0x7f69c40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 1 -- 172.21.15.37:0/3102317823 >> v1:172.21.15.37:6789/0 conn(0x7f69c41422b0 legacy=0x7f69c413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 10 monclient: _finish_hunting 0 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 1 monclient: found mon.noname-c 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 1 -- 172.21.15.37:0/3102317823 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c4151940 con 0x7f69c40b17d0 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 10 monclient: _finish_auth 0 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 10 monclient: _check_auth_tickets 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:06.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c8e96640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c8e96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 1 -- 172.21.15.37:0/3102317823 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f69b40017e0 con 0x7f69c40b17d0 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 10 monclient: dump: 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:06.364 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:06.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c8e96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:06.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 1 -- 172.21.15.37:0/3102317823 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69b4001a00 con 0x7f69c40b17d0 2024-03-29T11:38:06.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:06.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c8e96640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:06.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.358+0000 7f69c8e96640 4 set_mon_vals no callback set 2024-03-29T11:38:06.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.359+0000 7f69c8e96640 10 monclient: shutdown 2024-03-29T11:38:06.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.359+0000 7f69c8e96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:06.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.359+0000 7f69c8e96640 1 -- 172.21.15.37:0/3102317823 >> v1:172.21.15.146:6789/0 conn(0x7f69c40b17d0 legacy=0x7f69c40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.360+0000 7f69c8e96640 1 -- 172.21.15.37:0/3102317823 shutdown_connections 2024-03-29T11:38:06.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.360+0000 7f69c8e96640 1 -- 172.21.15.37:0/3102317823 >> 172.21.15.37:0/3102317823 conn(0x7f69c4066a00 msgr2=0x7f69c4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:06.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.361+0000 7f69c8e96640 1 -- 172.21.15.37:0/3102317823 shutdown_connections 2024-03-29T11:38:06.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.361+0000 7f69c8e96640 1 -- 172.21.15.37:0/3102317823 wait complete. 2024-03-29T11:38:06.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.362+0000 7f69c8e96640 10 monclient: build_initial_monmap 2024-03-29T11:38:06.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.362+0000 7f69c8e96640 10 monclient: monmap: 2024-03-29T11:38:06.367 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:06.367 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:06.367 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:38:06.367 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:38:06.367 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:06.367 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.363+0000 7f69c8e96640 1 librados: starting msgr at 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.363+0000 7f69c8e96640 1 librados: starting objecter 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.363+0000 7f69c8e96640 1 Processor -- start 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.363+0000 7f69c8e96640 1 -- start start 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.363+0000 7f69c8e96640 1 librados: setting wanted keys 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.363+0000 7f69c8e96640 1 librados: calling monclient init 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.363+0000 7f69c8e96640 10 monclient: init 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.364+0000 7f69c8e96640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.364+0000 7f69c8e96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.364+0000 7f69c8e96640 10 monclient(hunting): picked mon.noname-c con 0x7f69c40b17d0 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:06.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.364+0000 7f69c8e96640 10 monclient(hunting): picked mon.noname-a con 0x7f69c41422b0 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:06.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.364+0000 7f69c8e96640 10 monclient(hunting): picked mon.noname-b con 0x7f69c4142690 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:06.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.364+0000 7f69c8e96640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c4071620 con 0x7f69c41422b0 2024-03-29T11:38:06.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.364+0000 7f69c8e96640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c40ab9c0 con 0x7f69c4142690 2024-03-29T11:38:06.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.364+0000 7f69c8e96640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c414e410 con 0x7f69c40b17d0 2024-03-29T11:38:06.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.364+0000 7f69c8e96640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:06.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.364+0000 7f69c8e96640 10 monclient(hunting): authenticate will time out at 1065.813335s 2024-03-29T11:38:06.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.367+0000 7f69c259c640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f69c40b17d0 0x7f69c40709f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:34653/0 (socket says 172.21.15.37:34653) 2024-03-29T11:38:06.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.367+0000 7f69c259c640 1 -- 172.21.15.37:0/1429530761 learned_addr learned my addr 172.21.15.37:0/1429530761 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:06.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.368+0000 7f69b2ffd640 10 client.?.objecter ms_handle_connect 0x7f69c41422b0 2024-03-29T11:38:06.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.368+0000 7f69b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:06.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.369+0000 7f69b2ffd640 10 client.?.objecter ms_handle_connect 0x7f69c40b17d0 2024-03-29T11:38:06.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.369+0000 7f69b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:06.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.369+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219526577 0 0) 0x7f69c4071620 con 0x7f69c41422b0 2024-03-29T11:38:06.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.369+0000 7f69b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:06.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.369+0000 7f69b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:06.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.369+0000 7f69b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:06.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.371+0000 7f69b2ffd640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:38:06.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.371+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69a4004eb0 con 0x7f69c41422b0 2024-03-29T11:38:06.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.371+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189857461 0 0) 0x7f69c414e410 con 0x7f69c40b17d0 2024-03-29T11:38:06.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.371+0000 7f69b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:06.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.371+0000 7f69b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:06.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.371+0000 7f69b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:06.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.371+0000 7f69b2ffd640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:38:06.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.372+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c4071620 con 0x7f69c40b17d0 2024-03-29T11:38:06.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.372+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1483358231 0 0) 0x7f69a4004eb0 con 0x7f69c41422b0 2024-03-29T11:38:06.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.372+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69c414e410 con 0x7f69c41422b0 2024-03-29T11:38:06.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.372+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f69bc001830 con 0x7f69c41422b0 2024-03-29T11:38:06.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.372+0000 7f69b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:06.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.372+0000 7f69b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.373+0000 7f69b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.373+0000 7f69b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.373+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 281066013 0 0) 0x7f69c4071620 con 0x7f69c40b17d0 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.373+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69a4004eb0 con 0x7f69c40b17d0 2024-03-29T11:38:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.373+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f69b80019c0 con 0x7f69c40b17d0 2024-03-29T11:38:06.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.373+0000 7f69b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:06.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.373+0000 7f69b2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:06.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.373+0000 7f69b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:06.378 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:06.378 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:06.378 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:06.378 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:06.378 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:06.378 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:06.378 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:06.378 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:06.378 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:06.378 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:06.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.373+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3993619719 0 0) 0x7f69a4004eb0 con 0x7f69c40b17d0 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.373+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 >> v1:172.21.15.146:6789/0 conn(0x7f69c4142690 legacy=0x7f69c41eecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.373+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 >> v1:172.21.15.37:6789/0 conn(0x7f69c41422b0 legacy=0x7f69c412b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.373+0000 7f69b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.373+0000 7f69b2ffd640 1 monclient: found mon.c 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.373+0000 7f69b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69b2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c4071100 con 0x7f69c40b17d0 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69c8e96640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69c8e96640 10 monclient: _renew_subs 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69c8e96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69c8e96640 1 -- 172.21.15.37:0/1429530761 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c40713b0 con 0x7f69c40b17d0 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69c8e96640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:06.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69c8e96640 10 monclient: _renew_subs 2024-03-29T11:38:06.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69c8e96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:06.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69c8e96640 1 -- 172.21.15.37:0/1429530761 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c412bf20 con 0x7f69c40b17d0 2024-03-29T11:38:06.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69c8e96640 1 librados: init done 2024-03-29T11:38:06.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69b8001ca0 con 0x7f69c40b17d0 2024-03-29T11:38:06.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:06.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f69b8001b20 con 0x7f69c40b17d0 2024-03-29T11:38:06.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.374+0000 7f69b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:06.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.375+0000 7f69b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:06.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.375+0000 7f69b2ffd640 10 monclient: dump: 2024-03-29T11:38:06.380 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:06.380 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:06.380 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:06.380 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:06.381 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:06.381 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:06.381 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:06.381 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:06.381 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:06.381 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:06.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.375+0000 7f69b3fff640 4 set_mon_vals no callback set 2024-03-29T11:38:06.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.375+0000 7f69c8e96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:06.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.376+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f69b8003a60 con 0x7f69c40b17d0 2024-03-29T11:38:06.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.377+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f69b8084b20 con 0x7f69c40b17d0 2024-03-29T11:38:06.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.377+0000 7f69b2ffd640 10 client.4349.objecter ms_dispatch 0x7f69c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:06.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.377+0000 7f69b2ffd640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:06.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.377+0000 7f69b2ffd640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:06.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.377+0000 7f69b2ffd640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-29T11:38:06.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.378+0000 7f69c8e96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:06.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.378+0000 7f69c8e96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:06.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.378+0000 7f69c8e96640 1 -- 172.21.15.37:0/1429530761 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a8005080 con 0x7f69c40b17d0 2024-03-29T11:38:06.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.380+0000 7f69c1d9b640 10 monclient: get_auth_request con 0x7f69a406de90 auth_method 0 2024-03-29T11:38:06.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.380+0000 7f69b2ffd640 10 client.4349.objecter ms_handle_connect 0x7f69a406de90 2024-03-29T11:38:06.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.388+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69b8050230 con 0x7f69c40b17d0 2024-03-29T11:38:06.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.389+0000 7f69b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:06.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.389+0000 7f69b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:06.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.720+0000 7f69c8e96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:06.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.720+0000 7f69c8e96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:06.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.720+0000 7f69c8e96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:06.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.720+0000 7f69c8e96640 1 -- 172.21.15.37:0/1429530761 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69a8005330 con 0x7f69c40b17d0 2024-03-29T11:38:06.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.722+0000 7f69b2ffd640 1 -- 172.21.15.37:0/1429530761 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f69b8052330 con 0x7f69c40b17d0 2024-03-29T11:38:06.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.722+0000 7f69b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:06.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.722+0000 7f69b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:06.723 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:38:06.723 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:06.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.725+0000 7f69c8e96640 10 librados: watch_flush enter 2024-03-29T11:38:06.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.726+0000 7f69c8e96640 10 librados: watch_flush exit 2024-03-29T11:38:06.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.726+0000 7f69c8e96640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-29T11:38:06.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.726+0000 7f69c8e96640 10 client.4349.objecter successfully canceled tick 2024-03-29T11:38:06.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.727+0000 7f69c8e96640 1 -- 172.21.15.37:0/1429530761 >> v1:172.21.15.146:6816/2899149139 conn(0x7f69a406de90 legacy=0x7f69a4070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.727+0000 7f69c8e96640 10 monclient: shutdown 2024-03-29T11:38:06.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.727+0000 7f69c8e96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:06.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.727+0000 7f69c8e96640 1 -- 172.21.15.37:0/1429530761 >> v1:172.21.15.37:6790/0 conn(0x7f69c40b17d0 legacy=0x7f69c40709f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.727+0000 7f69c8e96640 1 -- 172.21.15.37:0/1429530761 shutdown_connections 2024-03-29T11:38:06.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.727+0000 7f69c8e96640 1 -- 172.21.15.37:0/1429530761 >> 172.21.15.37:0/1429530761 conn(0x7f69c4066a00 msgr2=0x7f69c413bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:06.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.727+0000 7f69c8e96640 1 -- 172.21.15.37:0/1429530761 shutdown_connections 2024-03-29T11:38:06.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.727+0000 7f69c8e96640 1 -- 172.21.15.37:0/1429530761 wait complete. 2024-03-29T11:38:06.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:06.728+0000 7f69c8e96640 1 librados: shutdown 2024-03-29T11:38:06.745 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:13.552 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:13.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.718+0000 7f2ee9715640 10 monclient: get_monmap_and_config 2024-03-29T11:38:13.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.718+0000 7f2ee9715640 10 monclient: build_initial_monmap 2024-03-29T11:38:13.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.718+0000 7f2ee9715640 10 monclient: monmap: 2024-03-29T11:38:13.719 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:13.719 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:13.719 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:38:13.719529+0000 2024-03-29T11:38:13.719 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:38:13.719529+0000 2024-03-29T11:38:13.720 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:13.720 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:13.720 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:13.720 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:38:13.720 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:38:13.720 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:13.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.722+0000 7f2ee9715640 1 Processor -- start 2024-03-29T11:38:13.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.722+0000 7f2ee9715640 1 -- start start 2024-03-29T11:38:13.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.722+0000 7f2ee9715640 10 monclient: init 2024-03-29T11:38:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.723+0000 7f2ee9715640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.723+0000 7f2ee9715640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.723+0000 7f2ee9715640 10 monclient(hunting): picked mon.noname-b con 0x7f2edc006890 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.723+0000 7f2ee9715640 10 monclient(hunting): picked mon.noname-a con 0x7f2edc006c70 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.723+0000 7f2ee9715640 10 monclient(hunting): picked mon.noname-c con 0x7f2edc02b4c0 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.723+0000 7f2ee9715640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc02c830 con 0x7f2edc006c70 2024-03-29T11:38:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.723+0000 7f2ee9715640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc02afa0 con 0x7f2edc02b4c0 2024-03-29T11:38:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.723+0000 7f2ee9715640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc02aba0 con 0x7f2edc006890 2024-03-29T11:38:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.723+0000 7f2ee9715640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:13.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.724+0000 7f2ee9715640 10 monclient(hunting): authenticate will time out at 1073.172937s 2024-03-29T11:38:13.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.724+0000 7f2ee3fff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f2edc006890 0x7f2edc0c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:44106/0 (socket says 172.21.15.37:44106) 2024-03-29T11:38:13.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.724+0000 7f2ee3fff640 1 -- 172.21.15.37:0/353190951 learned_addr learned my addr 172.21.15.37:0/353190951 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:13.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.725+0000 7f2ee2ffd640 1 -- 172.21.15.37:0/353190951 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701385264 0 0) 0x7f2edc02aba0 con 0x7f2edc006890 2024-03-29T11:38:13.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.725+0000 7f2ee2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:13.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.725+0000 7f2ee2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:13.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.725+0000 7f2ee2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:13.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.725+0000 7f2ee2ffd640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:38:13.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.725+0000 7f2ee2ffd640 1 -- 172.21.15.37:0/353190951 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed4003520 con 0x7f2edc006890 2024-03-29T11:38:13.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 1 -- 172.21.15.37:0/353190951 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2559189871 0 0) 0x7f2ed4003520 con 0x7f2edc006890 2024-03-29T11:38:13.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 1 -- 172.21.15.37:0/353190951 >> v1:172.21.15.146:6789/0 conn(0x7f2edc02b4c0 legacy=0x7f2edc0cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:13.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 1 -- 172.21.15.37:0/353190951 >> v1:172.21.15.37:6789/0 conn(0x7f2edc006c70 legacy=0x7f2edc0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:13.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:13.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:38:13.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:13.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:38:13.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 1 -- 172.21.15.37:0/353190951 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc02ad30 con 0x7f2edc006890 2024-03-29T11:38:13.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:13.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:13.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:13.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee9715640 5 monclient: authenticate success, global_id 4352 2024-03-29T11:38:13.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee9715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:13.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 1 -- 172.21.15.37:0/353190951 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2ee404f730 con 0x7f2edc006890 2024-03-29T11:38:13.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:13.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:13.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee2ffd640 10 monclient: dump: 2024-03-29T11:38:13.730 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:13.730 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.728+0000 7f2ee9715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.729+0000 7f2ee2ffd640 1 -- 172.21.15.37:0/353190951 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ee40008d0 con 0x7f2edc006890 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.729+0000 7f2ee2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.729+0000 7f2ee9715640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.729+0000 7f2ee9715640 4 set_mon_vals no callback set 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.729+0000 7f2ee2ffd640 1 -- 172.21.15.37:0/353190951 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2ee40758b0 con 0x7f2edc006890 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.729+0000 7f2ee2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:13.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.729+0000 7f2ee2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:13.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.729+0000 7f2ee2ffd640 10 monclient: dump: 2024-03-29T11:38:13.732 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:13.732 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:13.732 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:13.732 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:13.732 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:13.732 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:13.732 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:13.732 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:13.732 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:13.732 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:13.733 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.730+0000 7f2ee9715640 10 monclient: shutdown 2024-03-29T11:38:13.733 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.730+0000 7f2ee9715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:13.733 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.730+0000 7f2ee9715640 1 -- 172.21.15.37:0/353190951 >> v1:172.21.15.37:6790/0 conn(0x7f2edc006890 legacy=0x7f2edc0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:13.733 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.732+0000 7f2ee9715640 1 -- 172.21.15.37:0/353190951 shutdown_connections 2024-03-29T11:38:13.733 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.732+0000 7f2ee9715640 1 -- 172.21.15.37:0/353190951 >> 172.21.15.37:0/353190951 conn(0x7f2edc0058f0 msgr2=0x7f2edc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:13.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.732+0000 7f2ee9715640 1 -- 172.21.15.37:0/353190951 shutdown_connections 2024-03-29T11:38:13.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.732+0000 7f2ee9715640 1 -- 172.21.15.37:0/353190951 wait complete. 2024-03-29T11:38:13.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.733+0000 7f2ee9715640 10 monclient: build_initial_monmap 2024-03-29T11:38:13.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.733+0000 7f2ee9715640 10 monclient: monmap: 2024-03-29T11:38:13.735 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:13.735 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:13.735 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:38:13.735 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:38:13.735 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:13.735 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:13.735 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:13.735 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:38:13.735 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:38:13.735 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:13.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.734+0000 7f2ee9715640 1 librados: starting msgr at 2024-03-29T11:38:13.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.734+0000 7f2ee9715640 1 librados: starting objecter 2024-03-29T11:38:13.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.734+0000 7f2ee9715640 1 Processor -- start 2024-03-29T11:38:13.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.735+0000 7f2ee9715640 1 -- start start 2024-03-29T11:38:13.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.735+0000 7f2ee9715640 1 librados: setting wanted keys 2024-03-29T11:38:13.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.735+0000 7f2ee9715640 1 librados: calling monclient init 2024-03-29T11:38:13.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.735+0000 7f2ee9715640 10 monclient: init 2024-03-29T11:38:13.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.735+0000 7f2ee9715640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:13.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.735+0000 7f2ee9715640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:13.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.735+0000 7f2ee9715640 10 monclient(hunting): picked mon.noname-b con 0x7f2edc006890 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:13.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.736+0000 7f2ee9715640 10 monclient(hunting): picked mon.noname-c con 0x7f2edc02b4c0 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:13.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.736+0000 7f2ee9715640 10 monclient(hunting): picked mon.noname-a con 0x7f2edc164d50 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:13.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.736+0000 7f2ee9715640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc1641b0 con 0x7f2edc164d50 2024-03-29T11:38:13.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.736+0000 7f2ee9715640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc02f8e0 con 0x7f2edc006890 2024-03-29T11:38:13.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.736+0000 7f2ee9715640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc0c2bd0 con 0x7f2edc02b4c0 2024-03-29T11:38:13.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.736+0000 7f2ee9715640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:13.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.736+0000 7f2ee9715640 10 monclient(hunting): authenticate will time out at 1073.185007s 2024-03-29T11:38:13.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.739+0000 7f2ee37fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f2edc02b4c0 0x7f2edc164640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:44670/0 (socket says 172.21.15.37:44670) 2024-03-29T11:38:13.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.739+0000 7f2ee37fe640 1 -- 172.21.15.37:0/3277655535 learned_addr learned my addr 172.21.15.37:0/3277655535 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:13.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 10 client.?.objecter ms_handle_connect 0x7f2edc02b4c0 2024-03-29T11:38:13.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:13.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 10 client.?.objecter ms_handle_connect 0x7f2edc164d50 2024-03-29T11:38:13.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:13.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284133680 0 0) 0x7f2edc0c2bd0 con 0x7f2edc02b4c0 2024-03-29T11:38:13.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:13.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:13.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:13.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:38:13.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec8004eb0 con 0x7f2edc02b4c0 2024-03-29T11:38:13.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 10 client.?.objecter ms_handle_connect 0x7f2edc006890 2024-03-29T11:38:13.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:13.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958732852 0 0) 0x7f2edc1641b0 con 0x7f2edc164d50 2024-03-29T11:38:13.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:13.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:13.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:13.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.740+0000 7f2ee0ff9640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:38:13.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.741+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2edc0c2bd0 con 0x7f2edc164d50 2024-03-29T11:38:13.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.741+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967199168 0 0) 0x7f2edc02f8e0 con 0x7f2edc006890 2024-03-29T11:38:13.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.741+0000 7f2ee0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:13.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.741+0000 7f2ee0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:13.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.741+0000 7f2ee0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:13.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.741+0000 7f2ee0ff9640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:38:13.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.741+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2edc1641b0 con 0x7f2edc006890 2024-03-29T11:38:13.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.741+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1408895410 0 0) 0x7f2ec8004eb0 con 0x7f2edc02b4c0 2024-03-29T11:38:13.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.741+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2edc02f8e0 con 0x7f2edc02b4c0 2024-03-29T11:38:13.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.741+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2ed8001950 con 0x7f2edc02b4c0 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.741+0000 7f2ee0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.741+0000 7f2ee0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.741+0000 7f2ee0ff9640 10 monclient(hunting): dump: 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.742+0000 7f2ee0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.742+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3564894424 0 0) 0x7f2edc1641b0 con 0x7f2edc006890 2024-03-29T11:38:13.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.742+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ec8004eb0 con 0x7f2edc006890 2024-03-29T11:38:13.746 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.742+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2ee404f400 con 0x7f2edc006890 2024-03-29T11:38:13.746 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.742+0000 7f2ee0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:13.746 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.742+0000 7f2ee0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:13.746 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.742+0000 7f2ee0ff9640 10 monclient(hunting): dump: 2024-03-29T11:38:13.746 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:13.746 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:13.746 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:13.746 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:13.746 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:13.746 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:13.746 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:13.746 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:13.746 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:13.746 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:13.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.742+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3630286252 0 0) 0x7f2edc0c2bd0 con 0x7f2edc164d50 2024-03-29T11:38:13.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.742+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2edc1641b0 con 0x7f2edc164d50 2024-03-29T11:38:13.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.742+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3174948853 0 0) 0x7f2edc02f8e0 con 0x7f2edc02b4c0 2024-03-29T11:38:13.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.742+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 >> v1:172.21.15.146:6789/0 conn(0x7f2edc006890 legacy=0x7f2edc02e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 >> v1:172.21.15.37:6789/0 conn(0x7f2edc164d50 legacy=0x7f2edc0313b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee0ff9640 1 monclient: found mon.c 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc02e810 con 0x7f2edc02b4c0 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee9715640 5 monclient: authenticate success, global_id 4355 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee9715640 10 monclient: _renew_subs 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee9715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee9715640 1 -- 172.21.15.37:0/3277655535 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2edc02d560 con 0x7f2edc02b4c0 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee9715640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:13.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee9715640 10 monclient: _renew_subs 2024-03-29T11:38:13.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee9715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:13.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee9715640 1 -- 172.21.15.37:0/3277655535 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc031ca0 con 0x7f2edc02b4c0 2024-03-29T11:38:13.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.743+0000 7f2ee9715640 1 librados: init done 2024-03-29T11:38:13.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.747+0000 7f2ee9715640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:13.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.747+0000 7f2ee9715640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:13.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.747+0000 7f2ee9715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:13.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.747+0000 7f2ee9715640 1 -- 172.21.15.37:0/3277655535 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2edc0be360 con 0x7f2edc02b4c0 2024-03-29T11:38:13.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.748+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ed8003610 con 0x7f2edc02b4c0 2024-03-29T11:38:13.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.748+0000 7f2ee0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.748+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2ed80039a0 con 0x7f2edc02b4c0 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.748+0000 7f2ee0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.748+0000 7f2ee0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.748+0000 7f2ee0ff9640 10 monclient: dump: 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:13.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.749+0000 7f2ee1ffb640 4 set_mon_vals no callback set 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.749+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f2ed8003b00 con 0x7f2edc02b4c0 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.750+0000 7f2ee3fff640 10 monclient: get_auth_request con 0x7f2ec806e040 auth_method 0 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.750+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f2ed8054630 con 0x7f2edc02b4c0 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.750+0000 7f2ee0ff9640 10 client.4355.objecter ms_dispatch 0x7f2ee4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.750+0000 7f2ee0ff9640 3 client.4355.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.750+0000 7f2ee0ff9640 3 client.4355.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.750+0000 7f2ee0ff9640 20 client.4355.objecter dump_active .. 0 homeless 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.751+0000 7f2ee0ff9640 10 client.4355.objecter ms_handle_connect 0x7f2ec806e040 2024-03-29T11:38:13.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.764+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ed80508d0 con 0x7f2edc02b4c0 2024-03-29T11:38:13.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.764+0000 7f2ee0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:13.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:13.764+0000 7f2ee0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:14.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.098+0000 7f2ee9715640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:14.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.098+0000 7f2ee9715640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:14.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.098+0000 7f2ee9715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:14.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.098+0000 7f2ee9715640 1 -- 172.21.15.37:0/3277655535 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2edc004cd0 con 0x7f2edc02b4c0 2024-03-29T11:38:14.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.101+0000 7f2ee0ff9640 1 -- 172.21.15.37:0/3277655535 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f2ed80508d0 con 0x7f2edc02b4c0 2024-03-29T11:38:14.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.101+0000 7f2ee0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:14.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.101+0000 7f2ee0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:14.104 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:38:14.104 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:14.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.107+0000 7f2ee9715640 10 librados: watch_flush enter 2024-03-29T11:38:14.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.107+0000 7f2ee9715640 10 librados: watch_flush exit 2024-03-29T11:38:14.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.107+0000 7f2ee9715640 20 client.4355.objecter shutdown clearing up homeless session... 2024-03-29T11:38:14.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.107+0000 7f2ee9715640 10 client.4355.objecter successfully canceled tick 2024-03-29T11:38:14.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.108+0000 7f2ee9715640 1 -- 172.21.15.37:0/3277655535 >> v1:172.21.15.146:6816/2899149139 conn(0x7f2ec806e040 legacy=0x7f2ec8070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.109+0000 7f2ee9715640 10 monclient: shutdown 2024-03-29T11:38:14.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.109+0000 7f2ee9715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:14.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.109+0000 7f2ee9715640 1 -- 172.21.15.37:0/3277655535 >> v1:172.21.15.37:6790/0 conn(0x7f2edc02b4c0 legacy=0x7f2edc164640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.109+0000 7f2ee9715640 1 -- 172.21.15.37:0/3277655535 shutdown_connections 2024-03-29T11:38:14.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.109+0000 7f2ee9715640 1 -- 172.21.15.37:0/3277655535 >> 172.21.15.37:0/3277655535 conn(0x7f2edc0058f0 msgr2=0x7f2edc038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:14.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.109+0000 7f2ee9715640 1 -- 172.21.15.37:0/3277655535 shutdown_connections 2024-03-29T11:38:14.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.109+0000 7f2ee9715640 1 -- 172.21.15.37:0/3277655535 wait complete. 2024-03-29T11:38:14.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:14.109+0000 7f2ee9715640 1 librados: shutdown 2024-03-29T11:38:14.125 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:20.933 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:21.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.081+0000 7fc963496640 10 monclient: get_monmap_and_config 2024-03-29T11:38:21.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.081+0000 7fc963496640 10 monclient: build_initial_monmap 2024-03-29T11:38:21.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.081+0000 7fc963496640 10 monclient: monmap: 2024-03-29T11:38:21.082 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:21.082 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:21.082 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:38:21.082140+0000 2024-03-29T11:38:21.082 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:38:21.082140+0000 2024-03-29T11:38:21.082 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:21.082 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:21.082 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:21.082 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:38:21.082 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:38:21.082 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:21.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.082+0000 7fc963496640 1 Processor -- start 2024-03-29T11:38:21.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.082+0000 7fc963496640 1 -- start start 2024-03-29T11:38:21.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.082+0000 7fc963496640 10 monclient: init 2024-03-29T11:38:21.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.083+0000 7fc963496640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:21.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.083+0000 7fc963496640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:21.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.083+0000 7fc963496640 10 monclient(hunting): picked mon.noname-a con 0x7fc95c1422b0 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:21.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.083+0000 7fc963496640 10 monclient(hunting): picked mon.noname-c con 0x7fc95c142690 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:21.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.083+0000 7fc963496640 10 monclient(hunting): picked mon.noname-b con 0x7fc95c0b17d0 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:21.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.083+0000 7fc963496640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c142c80 con 0x7fc95c1422b0 2024-03-29T11:38:21.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.083+0000 7fc963496640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c0aae40 con 0x7fc95c142690 2024-03-29T11:38:21.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.083+0000 7fc963496640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c13fbd0 con 0x7fc95c0b17d0 2024-03-29T11:38:21.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.083+0000 7fc963496640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:21.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.083+0000 7fc963496640 10 monclient(hunting): authenticate will time out at 1080.532581s 2024-03-29T11:38:21.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.083+0000 7fc961232640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc95c1422b0 0x7fc95c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43756/0 (socket says 172.21.15.37:43756) 2024-03-29T11:38:21.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.083+0000 7fc961232640 1 -- 172.21.15.37:0/3086741648 learned_addr learned my addr 172.21.15.37:0/3086741648 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:21.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.084+0000 7fc953fff640 1 -- 172.21.15.37:0/3086741648 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314062137 0 0) 0x7fc95c0aae40 con 0x7fc95c142690 2024-03-29T11:38:21.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.084+0000 7fc953fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.084+0000 7fc953fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.084+0000 7fc953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.084+0000 7fc953fff640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:38:21.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.084+0000 7fc953fff640 1 -- 172.21.15.37:0/3086741648 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc948003520 con 0x7fc95c142690 2024-03-29T11:38:21.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 1 -- 172.21.15.37:0/3086741648 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931866747 0 0) 0x7fc95c142c80 con 0x7fc95c1422b0 2024-03-29T11:38:21.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:38:21.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 1 -- 172.21.15.37:0/3086741648 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc95c0aae40 con 0x7fc95c1422b0 2024-03-29T11:38:21.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 1 -- 172.21.15.37:0/3086741648 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3976238342 0 0) 0x7fc948003520 con 0x7fc95c142690 2024-03-29T11:38:21.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 1 -- 172.21.15.37:0/3086741648 >> v1:172.21.15.37:6790/0 conn(0x7fc95c0b17d0 legacy=0x7fc95c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:21.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 1 -- 172.21.15.37:0/3086741648 >> v1:172.21.15.37:6789/0 conn(0x7fc95c1422b0 legacy=0x7fc95c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:21.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 1 monclient: found mon.noname-c 2024-03-29T11:38:21.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:21.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:38:21.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 1 -- 172.21.15.37:0/3086741648 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95c151940 con 0x7fc95c142690 2024-03-29T11:38:21.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:21.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:21.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:21.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc963496640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:38:21.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 1 -- 172.21.15.37:0/3086741648 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc958001690 con 0x7fc95c142690 2024-03-29T11:38:21.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc963496640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:21.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc953fff640 10 monclient: dump: 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.085+0000 7fc963496640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.086+0000 7fc953fff640 1 -- 172.21.15.37:0/3086741648 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9580017f0 con 0x7fc95c142690 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.086+0000 7fc953fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.086+0000 7fc953fff640 1 -- 172.21.15.37:0/3086741648 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc958001b80 con 0x7fc95c142690 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.086+0000 7fc953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.086+0000 7fc953fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.086+0000 7fc953fff640 10 monclient: dump: 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.086+0000 7fc963496640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.086+0000 7fc963496640 4 set_mon_vals no callback set 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.087+0000 7fc963496640 10 monclient: shutdown 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.087+0000 7fc963496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.087+0000 7fc963496640 1 -- 172.21.15.37:0/3086741648 >> v1:172.21.15.146:6789/0 conn(0x7fc95c142690 legacy=0x7fc95c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.087+0000 7fc963496640 1 -- 172.21.15.37:0/3086741648 shutdown_connections 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.087+0000 7fc963496640 1 -- 172.21.15.37:0/3086741648 >> 172.21.15.37:0/3086741648 conn(0x7fc95c066a00 msgr2=0x7fc95c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.087+0000 7fc963496640 1 -- 172.21.15.37:0/3086741648 shutdown_connections 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.088+0000 7fc963496640 1 -- 172.21.15.37:0/3086741648 wait complete. 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.089+0000 7fc963496640 10 monclient: build_initial_monmap 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.089+0000 7fc963496640 10 monclient: monmap: 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.089+0000 7fc963496640 1 librados: starting msgr at 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.089+0000 7fc963496640 1 librados: starting objecter 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.089+0000 7fc963496640 1 Processor -- start 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.090+0000 7fc963496640 1 -- start start 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.090+0000 7fc963496640 1 librados: setting wanted keys 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.090+0000 7fc963496640 1 librados: calling monclient init 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.090+0000 7fc963496640 10 monclient: init 2024-03-29T11:38:21.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.090+0000 7fc963496640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:21.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.090+0000 7fc963496640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:21.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc963496640 10 monclient(hunting): picked mon.noname-c con 0x7fc95c0b17d0 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:21.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc963496640 10 monclient(hunting): picked mon.noname-b con 0x7fc95c1422b0 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:21.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc963496640 10 monclient(hunting): picked mon.noname-a con 0x7fc95c142690 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:21.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc961a33640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc95c142690 0x7fc95c1e72f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43946/0 (socket says 172.21.15.37:43946) 2024-03-29T11:38:21.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc961a33640 1 -- 172.21.15.37:0/898803333 learned_addr learned my addr 172.21.15.37:0/898803333 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:21.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc963496640 1 -- 172.21.15.37:0/898803333 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c06f310 con 0x7fc95c142690 2024-03-29T11:38:21.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc951ffb640 10 client.?.objecter ms_handle_connect 0x7fc95c142690 2024-03-29T11:38:21.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc951ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:21.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc951ffb640 10 client.?.objecter ms_handle_connect 0x7fc95c1422b0 2024-03-29T11:38:21.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc951ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:21.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc963496640 1 -- 172.21.15.37:0/898803333 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c12f350 con 0x7fc95c1422b0 2024-03-29T11:38:21.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc963496640 1 -- 172.21.15.37:0/898803333 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c12f4e0 con 0x7fc95c0b17d0 2024-03-29T11:38:21.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc951ffb640 10 client.?.objecter ms_handle_connect 0x7fc95c0b17d0 2024-03-29T11:38:21.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc951ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:21.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc963496640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:21.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.091+0000 7fc963496640 10 monclient(hunting): authenticate will time out at 1080.540757s 2024-03-29T11:38:21.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018482788 0 0) 0x7fc95c12f350 con 0x7fc95c1422b0 2024-03-29T11:38:21.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc940004eb0 con 0x7fc95c1422b0 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211303276 0 0) 0x7fc95c06f310 con 0x7fc95c142690 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc95c12f350 con 0x7fc95c142690 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526682058 0 0) 0x7fc95c12f4e0 con 0x7fc95c0b17d0 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.092+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc95c06f310 con 0x7fc95c0b17d0 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 47084176 0 0) 0x7fc940004eb0 con 0x7fc95c1422b0 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc95c12f4e0 con 0x7fc95c1422b0 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc958001b00 con 0x7fc95c1422b0 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 598405440 0 0) 0x7fc95c12f350 con 0x7fc95c142690 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc940004eb0 con 0x7fc95c142690 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc954004300 con 0x7fc95c142690 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2126702272 0 0) 0x7fc95c06f310 con 0x7fc95c0b17d0 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc95c12f350 con 0x7fc95c0b17d0 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc94c001930 con 0x7fc95c0b17d0 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3845586032 0 0) 0x7fc95c12f4e0 con 0x7fc95c1422b0 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 >> v1:172.21.15.37:6790/0 conn(0x7fc95c0b17d0 legacy=0x7fc95c1defb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 >> v1:172.21.15.37:6789/0 conn(0x7fc95c142690 legacy=0x7fc95c1e72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 1 monclient: found mon.b 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95c12f670 con 0x7fc95c1422b0 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc951ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc963496640 5 monclient: authenticate success, global_id 4336 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc963496640 10 monclient: _renew_subs 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc963496640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc963496640 1 -- 172.21.15.37:0/898803333 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95c0b6f50 con 0x7fc95c1422b0 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc963496640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc963496640 10 monclient: _renew_subs 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc963496640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc963496640 1 -- 172.21.15.37:0/898803333 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95c132700 con 0x7fc95c1422b0 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.093+0000 7fc963496640 1 librados: init done 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.094+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc958001d40 con 0x7fc95c1422b0 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.094+0000 7fc951ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.094+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc958003a40 con 0x7fc95c1422b0 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.094+0000 7fc951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.094+0000 7fc951ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.094+0000 7fc951ffb640 10 monclient: dump: 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.094+0000 7fc9537fe640 4 set_mon_vals no callback set 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.095+0000 7fc963496640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.095+0000 7fc963496640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.095+0000 7fc963496640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.096+0000 7fc963496640 1 -- 172.21.15.37:0/898803333 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93c005080 con 0x7fc95c1422b0 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.097+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fc958019a10 con 0x7fc95c1422b0 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.097+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fc958083820 con 0x7fc95c1422b0 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.097+0000 7fc951ffb640 10 client.4336.objecter ms_dispatch 0x7fc95c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.097+0000 7fc951ffb640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.097+0000 7fc951ffb640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.097+0000 7fc951ffb640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.097+0000 7fc961232640 10 monclient: get_auth_request con 0x7fc94006dfe0 auth_method 0 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.098+0000 7fc951ffb640 10 client.4336.objecter ms_handle_connect 0x7fc94006dfe0 2024-03-29T11:38:21.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.104+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc95804f100 con 0x7fc95c1422b0 2024-03-29T11:38:21.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.104+0000 7fc951ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:21.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.104+0000 7fc951ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:21.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.514+0000 7fc963496640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:21.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.514+0000 7fc963496640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:21.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.514+0000 7fc963496640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:38:21.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.514+0000 7fc963496640 1 -- 172.21.15.37:0/898803333 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc93c005330 con 0x7fc95c1422b0 2024-03-29T11:38:21.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.515+0000 7fc951ffb640 1 -- 172.21.15.37:0/898803333 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fc958051030 con 0x7fc95c1422b0 2024-03-29T11:38:21.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.515+0000 7fc951ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:21.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.515+0000 7fc951ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:21.516 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:38:21.516 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:21.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.520+0000 7fc963496640 10 librados: watch_flush enter 2024-03-29T11:38:21.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.520+0000 7fc963496640 10 librados: watch_flush exit 2024-03-29T11:38:21.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.520+0000 7fc963496640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-29T11:38:21.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.520+0000 7fc963496640 10 client.4336.objecter successfully canceled tick 2024-03-29T11:38:21.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.520+0000 7fc963496640 1 -- 172.21.15.37:0/898803333 >> v1:172.21.15.146:6816/2899149139 conn(0x7fc94006dfe0 legacy=0x7fc9400704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.520+0000 7fc963496640 10 monclient: shutdown 2024-03-29T11:38:21.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.520+0000 7fc963496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:21.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.520+0000 7fc963496640 1 -- 172.21.15.37:0/898803333 >> v1:172.21.15.146:6789/0 conn(0x7fc95c1422b0 legacy=0x7fc95c12ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.520+0000 7fc963496640 1 -- 172.21.15.37:0/898803333 shutdown_connections 2024-03-29T11:38:21.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.520+0000 7fc963496640 1 -- 172.21.15.37:0/898803333 >> 172.21.15.37:0/898803333 conn(0x7fc95c066a00 msgr2=0x7fc95c13c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:21.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.521+0000 7fc963496640 1 -- 172.21.15.37:0/898803333 shutdown_connections 2024-03-29T11:38:21.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.521+0000 7fc963496640 1 -- 172.21.15.37:0/898803333 wait complete. 2024-03-29T11:38:21.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:21.521+0000 7fc963496640 1 librados: shutdown 2024-03-29T11:38:21.537 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:28.343 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:28.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.487+0000 7f092359e640 10 monclient: get_monmap_and_config 2024-03-29T11:38:28.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.487+0000 7f092359e640 10 monclient: build_initial_monmap 2024-03-29T11:38:28.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.488+0000 7f092359e640 10 monclient: monmap: 2024-03-29T11:38:28.488 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:28.488 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:28.489 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:38:28.488853+0000 2024-03-29T11:38:28.489 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:38:28.488853+0000 2024-03-29T11:38:28.489 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:28.489 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:28.489 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:28.489 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:38:28.489 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:38:28.489 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:28.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.489+0000 7f092359e640 1 Processor -- start 2024-03-29T11:38:28.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.489+0000 7f092359e640 1 -- start start 2024-03-29T11:38:28.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.489+0000 7f092359e640 10 monclient: init 2024-03-29T11:38:28.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.490+0000 7f092359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:28.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.490+0000 7f092359e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:28.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.490+0000 7f092359e640 10 monclient(hunting): picked mon.noname-c con 0x7f091c0389e0 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:28.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.490+0000 7f092359e640 10 monclient(hunting): picked mon.noname-b con 0x7f091c038dc0 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:28.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.490+0000 7f092359e640 10 monclient(hunting): picked mon.noname-a con 0x7f091c030660 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:28.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.490+0000 7f092359e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c0c54c0 con 0x7f091c030660 2024-03-29T11:38:28.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.491+0000 7f092359e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c036430 con 0x7f091c0389e0 2024-03-29T11:38:28.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.491+0000 7f092359e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c039840 con 0x7f091c038dc0 2024-03-29T11:38:28.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.491+0000 7f092359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:28.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.491+0000 7f092359e640 10 monclient(hunting): authenticate will time out at 1087.939955s 2024-03-29T11:38:28.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.491+0000 7f0921d9b640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f091c038dc0 0x7f091c0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:58612/0 (socket says 172.21.15.37:58612) 2024-03-29T11:38:28.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.491+0000 7f0921d9b640 1 -- 172.21.15.37:0/4016954753 learned_addr learned my addr 172.21.15.37:0/4016954753 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:28.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 1 -- 172.21.15.37:0/4016954753 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293725579 0 0) 0x7f091c039840 con 0x7f091c038dc0 2024-03-29T11:38:28.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:38:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 1 -- 172.21.15.37:0/4016954753 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f090c003520 con 0x7f091c038dc0 2024-03-29T11:38:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 1 -- 172.21.15.37:0/4016954753 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558291351 0 0) 0x7f091c0c54c0 con 0x7f091c030660 2024-03-29T11:38:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:38:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 1 -- 172.21.15.37:0/4016954753 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f091c039840 con 0x7f091c030660 2024-03-29T11:38:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 1 -- 172.21.15.37:0/4016954753 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1090928385 0 0) 0x7f090c003520 con 0x7f091c038dc0 2024-03-29T11:38:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 1 -- 172.21.15.37:0/4016954753 >> v1:172.21.15.146:6789/0 conn(0x7f091c0389e0 legacy=0x7f091c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 1 -- 172.21.15.37:0/4016954753 >> v1:172.21.15.37:6789/0 conn(0x7f091c030660 legacy=0x7f091c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient: _finish_hunting 0 2024-03-29T11:38:28.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 1 monclient: found mon.noname-b 2024-03-29T11:38:28.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:28.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:38:28.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 1 -- 172.21.15.37:0/4016954753 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c035690 con 0x7f091c038dc0 2024-03-29T11:38:28.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient: _finish_auth 0 2024-03-29T11:38:28.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient: _check_auth_tickets 2024-03-29T11:38:28.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:28.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 1 -- 172.21.15.37:0/4016954753 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0918001690 con 0x7f091c038dc0 2024-03-29T11:38:28.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:28.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:28.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.492+0000 7f092159a640 10 monclient: dump: 2024-03-29T11:38:28.496 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:28.496 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:28.496 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:28.496 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:28.496 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:28.496 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:28.496 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:28.496 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:28.496 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:28.496 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:28.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.493+0000 7f092359e640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:38:28.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.493+0000 7f092359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:28.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.493+0000 7f092159a640 1 -- 172.21.15.37:0/4016954753 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0918001860 con 0x7f091c038dc0 2024-03-29T11:38:28.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.493+0000 7f092159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.493+0000 7f092159a640 1 -- 172.21.15.37:0/4016954753 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0918001c70 con 0x7f091c038dc0 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.493+0000 7f092159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.493+0000 7f092159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.493+0000 7f092159a640 10 monclient: dump: 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:28.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.493+0000 7f092359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:28.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.493+0000 7f092359e640 4 set_mon_vals no callback set 2024-03-29T11:38:28.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.494+0000 7f092359e640 10 monclient: shutdown 2024-03-29T11:38:28.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.494+0000 7f092359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:28.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.494+0000 7f092359e640 1 -- 172.21.15.37:0/4016954753 >> v1:172.21.15.37:6790/0 conn(0x7f091c038dc0 legacy=0x7f091c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.494+0000 7f092359e640 1 -- 172.21.15.37:0/4016954753 shutdown_connections 2024-03-29T11:38:28.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.494+0000 7f092359e640 1 -- 172.21.15.37:0/4016954753 >> 172.21.15.37:0/4016954753 conn(0x7f091c005c00 msgr2=0x7f091c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:28.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.494+0000 7f092359e640 1 -- 172.21.15.37:0/4016954753 shutdown_connections 2024-03-29T11:38:28.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.494+0000 7f092359e640 1 -- 172.21.15.37:0/4016954753 wait complete. 2024-03-29T11:38:28.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.495+0000 7f092359e640 10 monclient: build_initial_monmap 2024-03-29T11:38:28.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.495+0000 7f092359e640 10 monclient: monmap: 2024-03-29T11:38:28.506 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:28.506 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.496+0000 7f092359e640 1 librados: starting msgr at 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.496+0000 7f092359e640 1 librados: starting objecter 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.496+0000 7f092359e640 1 Processor -- start 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.496+0000 7f092359e640 1 -- start start 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.496+0000 7f092359e640 1 librados: setting wanted keys 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.496+0000 7f092359e640 1 librados: calling monclient init 2024-03-29T11:38:28.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.496+0000 7f092359e640 10 monclient: init 2024-03-29T11:38:28.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.497+0000 7f092359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:28.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.497+0000 7f092359e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:28.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.497+0000 7f092359e640 10 monclient(hunting): picked mon.noname-c con 0x7f091c030660 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:28.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.497+0000 7f092359e640 10 monclient(hunting): picked mon.noname-a con 0x7f091c0389e0 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:28.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.497+0000 7f092359e640 10 monclient(hunting): picked mon.noname-b con 0x7f091c02f840 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:28.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.497+0000 7f092359e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c0c7890 con 0x7f091c0389e0 2024-03-29T11:38:28.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.497+0000 7f092359e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c0c8430 con 0x7f091c02f840 2024-03-29T11:38:28.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.497+0000 7f092359e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c0c8740 con 0x7f091c030660 2024-03-29T11:38:28.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.497+0000 7f092359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:28.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.497+0000 7f092359e640 10 monclient(hunting): authenticate will time out at 1087.946693s 2024-03-29T11:38:28.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f092259c640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f091c030660 0x7f091c0c5650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:58834/0 (socket says 172.21.15.37:58834) 2024-03-29T11:38:28.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f092259c640 1 -- 172.21.15.37:0/154139018 learned_addr learned my addr 172.21.15.37:0/154139018 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:28.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 10 client.?.objecter ms_handle_connect 0x7f091c02f840 2024-03-29T11:38:28.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:28.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 10 client.?.objecter ms_handle_connect 0x7f091c030660 2024-03-29T11:38:28.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:28.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 10 client.?.objecter ms_handle_connect 0x7f091c0389e0 2024-03-29T11:38:28.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:28.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181015434 0 0) 0x7f091c0c8430 con 0x7f091c02f840 2024-03-29T11:38:28.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:28.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:28.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:28.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:38:28.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0904004eb0 con 0x7f091c02f840 2024-03-29T11:38:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385724130 0 0) 0x7f091c0c8740 con 0x7f091c030660 2024-03-29T11:38:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.498+0000 7f0912ffd640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:38:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f091c0c8430 con 0x7f091c030660 2024-03-29T11:38:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932998367 0 0) 0x7f091c0c7890 con 0x7f091c0389e0 2024-03-29T11:38:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:38:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f091c0c8740 con 0x7f091c0389e0 2024-03-29T11:38:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 618089747 0 0) 0x7f0904004eb0 con 0x7f091c02f840 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f091c0c7890 con 0x7f091c02f840 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f092404fdd0 con 0x7f091c02f840 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:28.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:28.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1208577726 0 0) 0x7f091c0c8430 con 0x7f091c030660 2024-03-29T11:38:28.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0904004eb0 con 0x7f091c030660 2024-03-29T11:38:28.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0914001990 con 0x7f091c030660 2024-03-29T11:38:28.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:28.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:28.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:28.512 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:28.512 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:28.512 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:28.512 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:28.512 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:28.512 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:28.512 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:28.512 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.499+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3143416530 0 0) 0x7f091c0c7890 con 0x7f091c02f840 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 >> v1:172.21.15.37:6790/0 conn(0x7f091c030660 legacy=0x7f091c0c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 >> v1:172.21.15.37:6789/0 conn(0x7f091c0389e0 legacy=0x7f091c0c05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f0912ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f0912ffd640 1 monclient: found mon.b 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f0912ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f0912ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c02fe20 con 0x7f091c02f840 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f0912ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f0912ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f0912ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f092359e640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:38:28.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f092359e640 10 monclient: _renew_subs 2024-03-29T11:38:28.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f092359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:38:28.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f092359e640 1 -- 172.21.15.37:0/154139018 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091c030070 con 0x7f091c02f840 2024-03-29T11:38:28.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f092359e640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:28.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f092359e640 10 monclient: _renew_subs 2024-03-29T11:38:28.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f092359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:38:28.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f092359e640 1 -- 172.21.15.37:0/154139018 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f091c1644f0 con 0x7f091c02f840 2024-03-29T11:38:28.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.500+0000 7f092359e640 1 librados: init done 2024-03-29T11:38:28.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.501+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f092404f400 con 0x7f091c02f840 2024-03-29T11:38:28.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.501+0000 7f0912ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:28.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.501+0000 7f0913fff640 4 set_mon_vals no callback set 2024-03-29T11:38:28.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.502+0000 7f092359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.502+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0924075f10 con 0x7f091c02f840 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.502+0000 7f0912ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.502+0000 7f0912ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.502+0000 7f0912ffd640 10 monclient: dump: 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.502+0000 7f092359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:28.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.503+0000 7f092359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:38:28.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.503+0000 7f092359e640 1 -- 172.21.15.37:0/154139018 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0908005080 con 0x7f091c02f840 2024-03-29T11:38:28.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.504+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f0924067bf0 con 0x7f091c02f840 2024-03-29T11:38:28.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.505+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f09240fe1e0 con 0x7f091c02f840 2024-03-29T11:38:28.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.505+0000 7f0912ffd640 10 client.4339.objecter ms_dispatch 0x7f0924000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:28.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.505+0000 7f0912ffd640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:28.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.505+0000 7f0912ffd640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:28.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.505+0000 7f0912ffd640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-29T11:38:28.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.505+0000 7f092259c640 10 monclient: get_auth_request con 0x7f090406db20 auth_method 0 2024-03-29T11:38:28.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.505+0000 7f0912ffd640 10 client.4339.objecter ms_handle_connect 0x7f090406db20 2024-03-29T11:38:28.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.513+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09240c9ac0 con 0x7f091c02f840 2024-03-29T11:38:28.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.513+0000 7f0912ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:28.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.513+0000 7f0912ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:28.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.893+0000 7f092359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:28.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.893+0000 7f092359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:28.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.893+0000 7f092359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:38:28.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.893+0000 7f092359e640 1 -- 172.21.15.37:0/154139018 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0908005330 con 0x7f091c02f840 2024-03-29T11:38:28.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.895+0000 7f0912ffd640 1 -- 172.21.15.37:0/154139018 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f09240cb9f0 con 0x7f091c02f840 2024-03-29T11:38:28.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.895+0000 7f0912ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:28.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.895+0000 7f0912ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:28.896 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:38:28.896 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:28.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.898+0000 7f092359e640 10 librados: watch_flush enter 2024-03-29T11:38:28.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.898+0000 7f092359e640 10 librados: watch_flush exit 2024-03-29T11:38:28.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.898+0000 7f092359e640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-29T11:38:28.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.899+0000 7f092359e640 10 client.4339.objecter successfully canceled tick 2024-03-29T11:38:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.899+0000 7f092359e640 1 -- 172.21.15.37:0/154139018 >> v1:172.21.15.146:6816/2899149139 conn(0x7f090406db20 legacy=0x7f090406ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.899+0000 7f092359e640 10 monclient: shutdown 2024-03-29T11:38:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.899+0000 7f092359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.899+0000 7f092359e640 1 -- 172.21.15.37:0/154139018 >> v1:172.21.15.146:6789/0 conn(0x7f091c02f840 legacy=0x7f091c0c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.899+0000 7f092359e640 1 -- 172.21.15.37:0/154139018 shutdown_connections 2024-03-29T11:38:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.899+0000 7f092359e640 1 -- 172.21.15.37:0/154139018 >> 172.21.15.37:0/154139018 conn(0x7f091c005c00 msgr2=0x7f091c0c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.899+0000 7f092359e640 1 -- 172.21.15.37:0/154139018 shutdown_connections 2024-03-29T11:38:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.899+0000 7f092359e640 1 -- 172.21.15.37:0/154139018 wait complete. 2024-03-29T11:38:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:28.899+0000 7f092359e640 1 librados: shutdown 2024-03-29T11:38:28.913 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:35.720 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:35.877 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.876+0000 7f0ee7fff640 10 monclient: get_monmap_and_config 2024-03-29T11:38:35.877 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.876+0000 7f0ee7fff640 10 monclient: build_initial_monmap 2024-03-29T11:38:35.877 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.876+0000 7f0ee7fff640 10 monclient: monmap: 2024-03-29T11:38:35.877 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:35.877 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:35.878 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:38:35.877475+0000 2024-03-29T11:38:35.878 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:38:35.877475+0000 2024-03-29T11:38:35.878 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:35.878 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:35.878 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:35.878 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:38:35.878 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:38:35.878 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:35.878 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.877+0000 7f0ee7fff640 1 Processor -- start 2024-03-29T11:38:35.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.877+0000 7f0ee7fff640 1 -- start start 2024-03-29T11:38:35.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.877+0000 7f0ee7fff640 10 monclient: init 2024-03-29T11:38:35.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.878+0000 7f0ee7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:35.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.878+0000 7f0ee7fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:35.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.878+0000 7f0ee7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f0ee0006b00 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:35.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.878+0000 7f0ee7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f0ee0033000 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:35.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.882+0000 7f0ee7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f0ee002c230 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:35.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.882+0000 7f0ee77fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f0ee002c230 0x7f0ee00cfd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:39857/0 (socket says 172.21.15.37:39857) 2024-03-29T11:38:35.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.882+0000 7f0ee77fe640 1 -- 172.21.15.37:0/1784158555 learned_addr learned my addr 172.21.15.37:0/1784158555 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:35.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.882+0000 7f0ee7fff640 1 -- 172.21.15.37:0/1784158555 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee002b7e0 con 0x7f0ee002c230 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.882+0000 7f0ee7fff640 1 -- 172.21.15.37:0/1784158555 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee002d210 con 0x7f0ee0033000 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.882+0000 7f0ee7fff640 1 -- 172.21.15.37:0/1784158555 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee002d4f0 con 0x7f0ee0006b00 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.882+0000 7f0ee5ffb640 1 -- 172.21.15.37:0/1784158555 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810668812 0 0) 0x7f0ee002b7e0 con 0x7f0ee002c230 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.882+0000 7f0ee7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.882+0000 7f0ee7fff640 10 monclient(hunting): authenticate will time out at 1095.331820s 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.882+0000 7f0ee5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.882+0000 7f0ee5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.883+0000 7f0ee5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.883+0000 7f0ee5ffb640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.883+0000 7f0ee5ffb640 1 -- 172.21.15.37:0/1784158555 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ecc003520 con 0x7f0ee002c230 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.883+0000 7f0ee5ffb640 1 -- 172.21.15.37:0/1784158555 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219308613 0 0) 0x7f0ee002d4f0 con 0x7f0ee0006b00 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.883+0000 7f0ee5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.883+0000 7f0ee5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.883+0000 7f0ee5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:35.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.883+0000 7f0ee5ffb640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.883+0000 7f0ee5ffb640 1 -- 172.21.15.37:0/1784158555 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee002b7e0 con 0x7f0ee0006b00 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.883+0000 7f0ee5ffb640 1 -- 172.21.15.37:0/1784158555 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2369927772 0 0) 0x7f0ecc003520 con 0x7f0ee002c230 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 1 -- 172.21.15.37:0/1784158555 >> v1:172.21.15.37:6790/0 conn(0x7f0ee0006b00 legacy=0x7f0ee00c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 1 -- 172.21.15.37:0/1784158555 >> v1:172.21.15.146:6789/0 conn(0x7f0ee0033000 legacy=0x7f0ee00c79f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 1 monclient: found mon.noname-a 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 1 -- 172.21.15.37:0/1784158555 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee002b2a0 con 0x7f0ee002c230 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 1 -- 172.21.15.37:0/1784158555 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0ee804f730 con 0x7f0ee002c230 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:35.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 10 monclient: dump: 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee7fff640 5 monclient: authenticate success, global_id 4356 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.884+0000 7f0ee5ffb640 1 -- 172.21.15.37:0/1784158555 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ee8075f10 con 0x7f0ee002c230 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.885+0000 7f0ee5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.885+0000 7f0ee5ffb640 1 -- 172.21.15.37:0/1784158555 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0ee80758b0 con 0x7f0ee002c230 2024-03-29T11:38:35.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.885+0000 7f0ee5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.885+0000 7f0ee5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.885+0000 7f0ee5ffb640 10 monclient: dump: 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.885+0000 7f0ee7fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.885+0000 7f0ee7fff640 4 set_mon_vals no callback set 2024-03-29T11:38:35.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.886+0000 7f0ee7fff640 10 monclient: shutdown 2024-03-29T11:38:35.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.886+0000 7f0ee7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:35.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.886+0000 7f0ee7fff640 1 -- 172.21.15.37:0/1784158555 >> v1:172.21.15.37:6789/0 conn(0x7f0ee002c230 legacy=0x7f0ee00cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:35.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.886+0000 7f0ee7fff640 1 -- 172.21.15.37:0/1784158555 shutdown_connections 2024-03-29T11:38:35.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.886+0000 7f0ee7fff640 1 -- 172.21.15.37:0/1784158555 >> 172.21.15.37:0/1784158555 conn(0x7f0ee00058f0 msgr2=0x7f0ee0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:35.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.887+0000 7f0ee7fff640 1 -- 172.21.15.37:0/1784158555 shutdown_connections 2024-03-29T11:38:35.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.887+0000 7f0ee7fff640 1 -- 172.21.15.37:0/1784158555 wait complete. 2024-03-29T11:38:35.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.889+0000 7f0ee7fff640 10 monclient: build_initial_monmap 2024-03-29T11:38:35.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.889+0000 7f0ee7fff640 10 monclient: monmap: 2024-03-29T11:38:35.893 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:35.893 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:35.893 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:38:35.893 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:38:35.893 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:35.893 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:35.893 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.889+0000 7f0ee7fff640 1 librados: starting msgr at 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.889+0000 7f0ee7fff640 1 librados: starting objecter 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.889+0000 7f0ee7fff640 1 Processor -- start 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.890+0000 7f0ee7fff640 1 -- start start 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.890+0000 7f0ee7fff640 1 librados: setting wanted keys 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.890+0000 7f0ee7fff640 1 librados: calling monclient init 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.890+0000 7f0ee7fff640 10 monclient: init 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.891+0000 7f0ee7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.891+0000 7f0ee7fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.892+0000 7f0ee7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f0ee0006b00 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.892+0000 7f0ee7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f0ee002c230 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:35.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.892+0000 7f0ee7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f0ee00c3c90 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:35.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.892+0000 7f0ee7fff640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee00c5310 con 0x7f0ee0006b00 2024-03-29T11:38:35.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.892+0000 7f0ee7fff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee00d20a0 con 0x7f0ee002c230 2024-03-29T11:38:35.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.892+0000 7f0ee7fff640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee00c7200 con 0x7f0ee00c3c90 2024-03-29T11:38:35.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.892+0000 7f0ee7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:35.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.892+0000 7f0ee7fff640 10 monclient(hunting): authenticate will time out at 1095.341280s 2024-03-29T11:38:35.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.893+0000 7f0ee77fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f0ee00c3c90 0x7f0ee016c4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:42644/0 (socket says 172.21.15.37:42644) 2024-03-29T11:38:35.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.893+0000 7f0ee77fe640 1 -- 172.21.15.37:0/3239613643 learned_addr learned my addr 172.21.15.37:0/3239613643 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:35.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.894+0000 7f0edffff640 10 client.?.objecter ms_handle_connect 0x7f0ee00c3c90 2024-03-29T11:38:35.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.894+0000 7f0edffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:35.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.894+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469045013 0 0) 0x7f0ee00c7200 con 0x7f0ee00c3c90 2024-03-29T11:38:35.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.894+0000 7f0edffff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:35.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.894+0000 7f0edffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:35.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.894+0000 7f0edffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:35.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.894+0000 7f0edffff640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:38:35.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.894+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ed0004eb0 con 0x7f0ee00c3c90 2024-03-29T11:38:35.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.895+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 62253594 0 0) 0x7f0ed0004eb0 con 0x7f0ee00c3c90 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.895+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ee00c7200 con 0x7f0ee00c3c90 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.895+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0ee804f400 con 0x7f0ee00c3c90 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.895+0000 7f0edffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.895+0000 7f0edffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.895+0000 7f0edffff640 10 monclient(hunting): dump: 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:35.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.895+0000 7f0edffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:35.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.895+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1760277953 0 0) 0x7f0ee00c7200 con 0x7f0ee00c3c90 2024-03-29T11:38:35.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.895+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 >> v1:172.21.15.146:6789/0 conn(0x7f0ee002c230 legacy=0x7f0ee00d2a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:35.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.897+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 >> v1:172.21.15.37:6789/0 conn(0x7f0ee0006b00 legacy=0x7f0ee00c4c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:35.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0edffff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:35.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0edffff640 1 monclient: found mon.c 2024-03-29T11:38:35.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0edffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:35.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0edffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:35.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee00c2b10 con 0x7f0ee00c3c90 2024-03-29T11:38:35.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0edffff640 10 monclient: _finish_auth 0 2024-03-29T11:38:35.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0edffff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:35.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0edffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:35.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0ee7fff640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:38:35.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0ee7fff640 10 monclient: _renew_subs 2024-03-29T11:38:35.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0ee7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:35.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0ee7fff640 1 -- 172.21.15.37:0/3239613643 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee00c2df0 con 0x7f0ee00c3c90 2024-03-29T11:38:35.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0ee7fff640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:35.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0ee7fff640 10 monclient: _renew_subs 2024-03-29T11:38:35.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0ee7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:35.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0ee7fff640 1 -- 172.21.15.37:0/3239613643 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee00c33b0 con 0x7f0ee00c3c90 2024-03-29T11:38:35.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.898+0000 7f0ee7fff640 1 librados: init done 2024-03-29T11:38:35.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.899+0000 7f0ee7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:35.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.899+0000 7f0ee7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:35.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.899+0000 7f0ee7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:35.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.899+0000 7f0ee7fff640 1 -- 172.21.15.37:0/3239613643 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee0004cd0 con 0x7f0ee00c3c90 2024-03-29T11:38:35.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.899+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ee804fdd0 con 0x7f0ee00c3c90 2024-03-29T11:38:35.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.899+0000 7f0edffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:35.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.900+0000 7f0ee57fa640 4 set_mon_vals no callback set 2024-03-29T11:38:35.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.901+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0ee8067bf0 con 0x7f0ee00c3c90 2024-03-29T11:38:35.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.901+0000 7f0edffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:35.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.901+0000 7f0edffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:35.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.901+0000 7f0edffff640 10 monclient: dump: 2024-03-29T11:38:35.902 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:35.902 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:35.902 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:35.902 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:35.902 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:35.902 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:35.902 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:35.902 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:35.903 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:35.903 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:35.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.901+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f0ee8067e10 con 0x7f0ee00c3c90 2024-03-29T11:38:35.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.902+0000 7f0ee6ffd640 10 monclient: get_auth_request con 0x7f0ed006dc80 auth_method 0 2024-03-29T11:38:35.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.902+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f0ee80ff7d0 con 0x7f0ee00c3c90 2024-03-29T11:38:35.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.902+0000 7f0edffff640 10 client.4370.objecter ms_dispatch 0x7f0ee8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:35.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.902+0000 7f0edffff640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:35.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.902+0000 7f0edffff640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:35.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.902+0000 7f0edffff640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-29T11:38:35.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.902+0000 7f0edffff640 10 client.4370.objecter ms_handle_connect 0x7f0ed006dc80 2024-03-29T11:38:35.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.908+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ee80cb130 con 0x7f0ee00c3c90 2024-03-29T11:38:35.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.908+0000 7f0edffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:35.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:35.908+0000 7f0edffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.239+0000 7f0ee7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.239+0000 7f0ee7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.239+0000 7f0ee7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.239+0000 7f0ee7fff640 1 -- 172.21.15.37:0/3239613643 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ee00086f0 con 0x7f0ee00c3c90 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.239+0000 7f0edffff640 1 -- 172.21.15.37:0/3239613643 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f0ee80cd060 con 0x7f0ee00c3c90 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.239+0000 7f0edffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.239+0000 7f0edffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:36.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.243+0000 7f0ee7fff640 10 librados: watch_flush enter 2024-03-29T11:38:36.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.243+0000 7f0ee7fff640 10 librados: watch_flush exit 2024-03-29T11:38:36.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.243+0000 7f0ee7fff640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-29T11:38:36.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.243+0000 7f0ee7fff640 10 client.4370.objecter successfully canceled tick 2024-03-29T11:38:36.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.243+0000 7f0ee7fff640 1 -- 172.21.15.37:0/3239613643 >> v1:172.21.15.146:6816/2899149139 conn(0x7f0ed006dc80 legacy=0x7f0ed0070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.243+0000 7f0ee7fff640 10 monclient: shutdown 2024-03-29T11:38:36.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.243+0000 7f0ee7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:36.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.243+0000 7f0ee7fff640 1 -- 172.21.15.37:0/3239613643 >> v1:172.21.15.37:6790/0 conn(0x7f0ee00c3c90 legacy=0x7f0ee016c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.243+0000 7f0ee7fff640 1 -- 172.21.15.37:0/3239613643 shutdown_connections 2024-03-29T11:38:36.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.243+0000 7f0ee7fff640 1 -- 172.21.15.37:0/3239613643 >> 172.21.15.37:0/3239613643 conn(0x7f0ee00058f0 msgr2=0x7f0ee0005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:36.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.243+0000 7f0ee7fff640 1 -- 172.21.15.37:0/3239613643 shutdown_connections 2024-03-29T11:38:36.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.243+0000 7f0ee7fff640 1 -- 172.21.15.37:0/3239613643 wait complete. 2024-03-29T11:38:36.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:36.243+0000 7f0ee7fff640 1 librados: shutdown 2024-03-29T11:38:36.258 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:43.063 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:43.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.208+0000 7f982af25640 10 monclient: get_monmap_and_config 2024-03-29T11:38:43.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.208+0000 7f982af25640 10 monclient: build_initial_monmap 2024-03-29T11:38:43.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.208+0000 7f982af25640 10 monclient: monmap: 2024-03-29T11:38:43.210 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:43.210 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:43.210 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:38:43.209066+0000 2024-03-29T11:38:43.211 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:38:43.209066+0000 2024-03-29T11:38:43.211 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:43.211 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:43.211 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:43.211 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:38:43.211 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:38:43.211 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:43.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.209+0000 7f982af25640 1 Processor -- start 2024-03-29T11:38:43.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.209+0000 7f982af25640 1 -- start start 2024-03-29T11:38:43.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.209+0000 7f982af25640 10 monclient: init 2024-03-29T11:38:43.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.210+0000 7f982af25640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:43.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.210+0000 7f982af25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:43.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.210+0000 7f982af25640 10 monclient(hunting): picked mon.noname-a con 0x7f98240b4100 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:43.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.210+0000 7f982af25640 10 monclient(hunting): picked mon.noname-b con 0x7f982413ee90 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:43.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.210+0000 7f982af25640 10 monclient(hunting): picked mon.noname-c con 0x7f98240af900 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:43.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.210+0000 7f982af25640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98240afd10 con 0x7f98240b4100 2024-03-29T11:38:43.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.210+0000 7f982af25640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98240af6f0 con 0x7f98240af900 2024-03-29T11:38:43.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.210+0000 7f982af25640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98240aaf60 con 0x7f982413ee90 2024-03-29T11:38:43.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.210+0000 7f982af25640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:43.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.210+0000 7f982af25640 10 monclient(hunting): authenticate will time out at 1102.659677s 2024-03-29T11:38:43.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.211+0000 7f9823fff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f982413ee90 0x7f98240adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:55284/0 (socket says 172.21.15.37:55284) 2024-03-29T11:38:43.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.211+0000 7f9823fff640 1 -- 172.21.15.37:0/2890844445 learned_addr learned my addr 172.21.15.37:0/2890844445 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:43.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.212+0000 7f98237fe640 1 -- 172.21.15.37:0/2890844445 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148705725 0 0) 0x7f98240aaf60 con 0x7f982413ee90 2024-03-29T11:38:43.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.212+0000 7f98237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:43.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.212+0000 7f98237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:43.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.212+0000 7f98237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:43.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.212+0000 7f98237fe640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:38:43.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.212+0000 7f98237fe640 1 -- 172.21.15.37:0/2890844445 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f980c003520 con 0x7f982413ee90 2024-03-29T11:38:43.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.213+0000 7f98237fe640 1 -- 172.21.15.37:0/2890844445 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4079679319 0 0) 0x7f980c003520 con 0x7f982413ee90 2024-03-29T11:38:43.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.213+0000 7f98237fe640 1 -- 172.21.15.37:0/2890844445 >> v1:172.21.15.146:6789/0 conn(0x7f98240af900 legacy=0x7f982413b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:43.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.215+0000 7f98237fe640 1 -- 172.21.15.37:0/2890844445 >> v1:172.21.15.37:6789/0 conn(0x7f98240b4100 legacy=0x7f98240a9850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:43.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.216+0000 7f98237fe640 10 monclient: _finish_hunting 0 2024-03-29T11:38:43.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.216+0000 7f98237fe640 1 monclient: found mon.noname-b 2024-03-29T11:38:43.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.216+0000 7f98237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:43.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.216+0000 7f98237fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:38:43.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.216+0000 7f98237fe640 1 -- 172.21.15.37:0/2890844445 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982413e970 con 0x7f982413ee90 2024-03-29T11:38:43.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.216+0000 7f98237fe640 10 monclient: _finish_auth 0 2024-03-29T11:38:43.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.216+0000 7f98237fe640 10 monclient: _check_auth_tickets 2024-03-29T11:38:43.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.216+0000 7f98237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:43.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.216+0000 7f98237fe640 1 -- 172.21.15.37:0/2890844445 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9818001690 con 0x7f982413ee90 2024-03-29T11:38:43.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.216+0000 7f98237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.216+0000 7f98237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.216+0000 7f98237fe640 10 monclient: dump: 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.216+0000 7f982af25640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.216+0000 7f982af25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.217+0000 7f98237fe640 1 -- 172.21.15.37:0/2890844445 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9818001860 con 0x7f982413ee90 2024-03-29T11:38:43.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.217+0000 7f98237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.217+0000 7f98237fe640 1 -- 172.21.15.37:0/2890844445 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9818001cb0 con 0x7f982413ee90 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.217+0000 7f98237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.217+0000 7f98237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.217+0000 7f98237fe640 10 monclient: dump: 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.217+0000 7f982af25640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:43.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.217+0000 7f982af25640 4 set_mon_vals no callback set 2024-03-29T11:38:43.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.218+0000 7f982af25640 10 monclient: shutdown 2024-03-29T11:38:43.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.218+0000 7f982af25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:43.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.218+0000 7f982af25640 1 -- 172.21.15.37:0/2890844445 >> v1:172.21.15.37:6790/0 conn(0x7f982413ee90 legacy=0x7f98240adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.218+0000 7f982af25640 1 -- 172.21.15.37:0/2890844445 shutdown_connections 2024-03-29T11:38:43.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.218+0000 7f982af25640 1 -- 172.21.15.37:0/2890844445 >> 172.21.15.37:0/2890844445 conn(0x7f98240b35e0 msgr2=0x7f98240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:43.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.218+0000 7f982af25640 1 -- 172.21.15.37:0/2890844445 shutdown_connections 2024-03-29T11:38:43.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.218+0000 7f982af25640 1 -- 172.21.15.37:0/2890844445 wait complete. 2024-03-29T11:38:43.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.219+0000 7f982af25640 10 monclient: build_initial_monmap 2024-03-29T11:38:43.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.219+0000 7f982af25640 10 monclient: monmap: 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.220+0000 7f982af25640 1 librados: starting msgr at 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.220+0000 7f982af25640 1 librados: starting objecter 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.220+0000 7f982af25640 1 Processor -- start 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.220+0000 7f982af25640 1 -- start start 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.220+0000 7f982af25640 1 librados: setting wanted keys 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.220+0000 7f982af25640 1 librados: calling monclient init 2024-03-29T11:38:43.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.220+0000 7f982af25640 10 monclient: init 2024-03-29T11:38:43.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.221+0000 7f982af25640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:43.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.221+0000 7f982af25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:43.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.221+0000 7f982af25640 10 monclient(hunting): picked mon.noname-b con 0x7f98240af900 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:43.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.221+0000 7f982af25640 10 monclient(hunting): picked mon.noname-a con 0x7f98240b4100 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:43.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.221+0000 7f982af25640 10 monclient(hunting): picked mon.noname-c con 0x7f982413ee90 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:43.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.221+0000 7f982af25640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9824071f70 con 0x7f98240b4100 2024-03-29T11:38:43.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.221+0000 7f982af25640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9824128910 con 0x7f98240af900 2024-03-29T11:38:43.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.221+0000 7f982af25640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9824071540 con 0x7f982413ee90 2024-03-29T11:38:43.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.221+0000 7f982af25640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:43.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.221+0000 7f982af25640 10 monclient(hunting): authenticate will time out at 1102.670179s 2024-03-29T11:38:43.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.224+0000 7f98294c2640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f982413ee90 0x7f9824086960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:56228/0 (socket says 172.21.15.37:56228) 2024-03-29T11:38:43.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.224+0000 7f98294c2640 1 -- 172.21.15.37:0/486812585 learned_addr learned my addr 172.21.15.37:0/486812585 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:43.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.224+0000 7f98217fa640 10 client.?.objecter ms_handle_connect 0x7f982413ee90 2024-03-29T11:38:43.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.224+0000 7f98217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:43.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.224+0000 7f98217fa640 10 client.?.objecter ms_handle_connect 0x7f98240af900 2024-03-29T11:38:43.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.224+0000 7f98217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:43.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.224+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025658404 0 0) 0x7f9824071540 con 0x7f982413ee90 2024-03-29T11:38:43.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.224+0000 7f98217fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:43.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.224+0000 7f98217fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:43.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.224+0000 7f98217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:43.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.224+0000 7f98217fa640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.225+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9810004eb0 con 0x7f982413ee90 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.225+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602683102 0 0) 0x7f9824128910 con 0x7f98240af900 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.225+0000 7f98217fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.225+0000 7f98217fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.225+0000 7f98217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.225+0000 7f98217fa640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.225+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9824071540 con 0x7f98240af900 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1705239584 0 0) 0x7f9810004eb0 con 0x7f982413ee90 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9824128910 con 0x7f982413ee90 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f98140017f0 con 0x7f982413ee90 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 10 monclient(hunting): dump: 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:43.229 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 10 client.?.objecter ms_handle_connect 0x7f98240b4100 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 669335489 0 0) 0x7f9824128910 con 0x7f982413ee90 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 >> v1:172.21.15.146:6789/0 conn(0x7f98240af900 legacy=0x7f982407e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 >> v1:172.21.15.37:6789/0 conn(0x7f98240b4100 legacy=0x7f9824126630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 10 monclient: _finish_hunting 0 2024-03-29T11:38:43.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 1 monclient: found mon.c 2024-03-29T11:38:43.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:43.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:43.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.226+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982407ecf0 con 0x7f982413ee90 2024-03-29T11:38:43.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.227+0000 7f98217fa640 10 monclient: _finish_auth 0 2024-03-29T11:38:43.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.227+0000 7f98217fa640 10 monclient: _check_auth_tickets 2024-03-29T11:38:43.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.227+0000 7f98217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:43.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.227+0000 7f982af25640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:38:43.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.227+0000 7f982af25640 10 monclient: _renew_subs 2024-03-29T11:38:43.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.227+0000 7f982af25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:43.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.227+0000 7f982af25640 1 -- 172.21.15.37:0/486812585 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9824072400 con 0x7f982413ee90 2024-03-29T11:38:43.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.227+0000 7f982af25640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:43.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.227+0000 7f982af25640 10 monclient: _renew_subs 2024-03-29T11:38:43.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.227+0000 7f982af25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:43.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.227+0000 7f982af25640 1 -- 172.21.15.37:0/486812585 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f982413de80 con 0x7f982413ee90 2024-03-29T11:38:43.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.227+0000 7f982af25640 1 librados: init done 2024-03-29T11:38:43.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.228+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9814001950 con 0x7f982413ee90 2024-03-29T11:38:43.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.228+0000 7f98217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:43.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.228+0000 7f9822ffd640 4 set_mon_vals no callback set 2024-03-29T11:38:43.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.229+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9814004aa0 con 0x7f982413ee90 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.229+0000 7f98217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.229+0000 7f98217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.229+0000 7f98217fa640 10 monclient: dump: 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.229+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f981401aa70 con 0x7f982413ee90 2024-03-29T11:38:43.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.230+0000 7f982af25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:43.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.230+0000 7f982af25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:43.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.230+0000 7f982af25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:43.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.230+0000 7f982af25640 1 -- 172.21.15.37:0/486812585 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9804005080 con 0x7f982413ee90 2024-03-29T11:38:43.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.231+0000 7f9828cc1640 10 monclient: get_auth_request con 0x7f981006dd70 auth_method 0 2024-03-29T11:38:43.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.231+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f9814084a00 con 0x7f982413ee90 2024-03-29T11:38:43.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.231+0000 7f98217fa640 10 client.4376.objecter ms_dispatch 0x7f9824000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:43.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.231+0000 7f98217fa640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:43.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.231+0000 7f98217fa640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:43.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.232+0000 7f98217fa640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-29T11:38:43.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.232+0000 7f98217fa640 10 client.4376.objecter ms_handle_connect 0x7f981006dd70 2024-03-29T11:38:43.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.237+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9814050360 con 0x7f982413ee90 2024-03-29T11:38:43.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.237+0000 7f98217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:43.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.237+0000 7f98217fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:43.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.563+0000 7f982af25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:43.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.563+0000 7f982af25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:43.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.563+0000 7f982af25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:43.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.563+0000 7f982af25640 1 -- 172.21.15.37:0/486812585 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9804005330 con 0x7f982413ee90 2024-03-29T11:38:43.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.563+0000 7f98217fa640 1 -- 172.21.15.37:0/486812585 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f9814052290 con 0x7f982413ee90 2024-03-29T11:38:43.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.563+0000 7f98217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:43.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.563+0000 7f98217fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:43.565 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:38:43.565 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:43.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.568+0000 7f982af25640 10 librados: watch_flush enter 2024-03-29T11:38:43.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.568+0000 7f982af25640 10 librados: watch_flush exit 2024-03-29T11:38:43.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.568+0000 7f982af25640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-29T11:38:43.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.568+0000 7f982af25640 10 client.4376.objecter successfully canceled tick 2024-03-29T11:38:43.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.568+0000 7f982af25640 1 -- 172.21.15.37:0/486812585 >> v1:172.21.15.146:6816/2899149139 conn(0x7f981006dd70 legacy=0x7f9810070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.568+0000 7f982af25640 10 monclient: shutdown 2024-03-29T11:38:43.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.568+0000 7f982af25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:43.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.568+0000 7f982af25640 1 -- 172.21.15.37:0/486812585 >> v1:172.21.15.37:6790/0 conn(0x7f982413ee90 legacy=0x7f9824086960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.569+0000 7f982af25640 1 -- 172.21.15.37:0/486812585 shutdown_connections 2024-03-29T11:38:43.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.569+0000 7f982af25640 1 -- 172.21.15.37:0/486812585 >> 172.21.15.37:0/486812585 conn(0x7f98240b35e0 msgr2=0x7f98240b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:43.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.569+0000 7f982af25640 1 -- 172.21.15.37:0/486812585 shutdown_connections 2024-03-29T11:38:43.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.569+0000 7f982af25640 1 -- 172.21.15.37:0/486812585 wait complete. 2024-03-29T11:38:43.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:43.569+0000 7f982af25640 1 librados: shutdown 2024-03-29T11:38:43.587 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:50.393 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:50.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.550+0000 7f8b03a27640 10 monclient: get_monmap_and_config 2024-03-29T11:38:50.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.550+0000 7f8b03a27640 10 monclient: build_initial_monmap 2024-03-29T11:38:50.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.550+0000 7f8b03a27640 10 monclient: monmap: 2024-03-29T11:38:50.551 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:50.551 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:50.551 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:38:50.551315+0000 2024-03-29T11:38:50.551 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:38:50.551315+0000 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:50.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.551+0000 7f8b03a27640 1 Processor -- start 2024-03-29T11:38:50.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.551+0000 7f8b03a27640 1 -- start start 2024-03-29T11:38:50.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.551+0000 7f8b03a27640 10 monclient: init 2024-03-29T11:38:50.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.552+0000 7f8b03a27640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:50.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.552+0000 7f8b03a27640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:50.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.555+0000 7f8b03a27640 10 monclient(hunting): picked mon.noname-c con 0x7f8afc1457f0 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:50.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.555+0000 7f8b03a27640 10 monclient(hunting): picked mon.noname-b con 0x7f8afc13f650 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:50.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.555+0000 7f8b03a27640 10 monclient(hunting): picked mon.noname-a con 0x7f8afc0a9d70 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:50.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.555+0000 7f8b03a27640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8afc0aaef0 con 0x7f8afc0a9d70 2024-03-29T11:38:50.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.555+0000 7f8b03a27640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8afc145bd0 con 0x7f8afc1457f0 2024-03-29T11:38:50.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.555+0000 7f8b03a27640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8afc145eb0 con 0x7f8afc13f650 2024-03-29T11:38:50.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.555+0000 7f8b03a27640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:50.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.555+0000 7f8b03a27640 10 monclient(hunting): authenticate will time out at 1110.004681s 2024-03-29T11:38:50.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.556+0000 7f8b017c3640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f8afc1457f0 0x7f8afc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:43315/0 (socket says 172.21.15.37:43315) 2024-03-29T11:38:50.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.556+0000 7f8b017c3640 1 -- 172.21.15.37:0/1048824903 learned_addr learned my addr 172.21.15.37:0/1048824903 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.558+0000 7f8af3fff640 1 -- 172.21.15.37:0/1048824903 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321389106 0 0) 0x7f8afc145bd0 con 0x7f8afc1457f0 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.558+0000 7f8af3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.558+0000 7f8af3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.558+0000 7f8af3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.558+0000 7f8af3fff640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.558+0000 7f8af3fff640 1 -- 172.21.15.37:0/1048824903 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ae8003520 con 0x7f8afc1457f0 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 1 -- 172.21.15.37:0/1048824903 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665582263 0 0) 0x7f8afc0aaef0 con 0x7f8afc0a9d70 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 1 -- 172.21.15.37:0/1048824903 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8afc145bd0 con 0x7f8afc0a9d70 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 1 -- 172.21.15.37:0/1048824903 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618699251 0 0) 0x7f8afc145eb0 con 0x7f8afc13f650 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 1 -- 172.21.15.37:0/1048824903 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8afc0aaef0 con 0x7f8afc13f650 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 1 -- 172.21.15.37:0/1048824903 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2385584649 0 0) 0x7f8ae8003520 con 0x7f8afc1457f0 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 1 -- 172.21.15.37:0/1048824903 >> v1:172.21.15.37:6790/0 conn(0x7f8afc13f650 legacy=0x7f8afc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 1 -- 172.21.15.37:0/1048824903 >> v1:172.21.15.37:6789/0 conn(0x7f8afc0a9d70 legacy=0x7f8afc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 1 monclient: found mon.noname-c 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 1 -- 172.21.15.37:0/1048824903 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8afc146340 con 0x7f8afc1457f0 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 1 -- 172.21.15.37:0/1048824903 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8af8001610 con 0x7f8afc1457f0 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8af3fff640 10 monclient: dump: 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:50.563 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:50.563 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:50.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8b03a27640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:38:50.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.559+0000 7f8b03a27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:50.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.560+0000 7f8af3fff640 1 -- 172.21.15.37:0/1048824903 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8af8001770 con 0x7f8afc1457f0 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.560+0000 7f8af3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.560+0000 7f8b03a27640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.560+0000 7f8af3fff640 1 -- 172.21.15.37:0/1048824903 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8af8001bc0 con 0x7f8afc1457f0 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.560+0000 7f8af3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.560+0000 7f8b03a27640 4 set_mon_vals no callback set 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.560+0000 7f8af3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.560+0000 7f8af3fff640 10 monclient: dump: 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.561+0000 7f8b03a27640 10 monclient: shutdown 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.561+0000 7f8b03a27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.561+0000 7f8b03a27640 1 -- 172.21.15.37:0/1048824903 >> v1:172.21.15.146:6789/0 conn(0x7f8afc1457f0 legacy=0x7f8afc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.561+0000 7f8b03a27640 1 -- 172.21.15.37:0/1048824903 shutdown_connections 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.561+0000 7f8b03a27640 1 -- 172.21.15.37:0/1048824903 >> 172.21.15.37:0/1048824903 conn(0x7f8afc066a00 msgr2=0x7f8afc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.562+0000 7f8b03a27640 1 -- 172.21.15.37:0/1048824903 shutdown_connections 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.562+0000 7f8b03a27640 1 -- 172.21.15.37:0/1048824903 wait complete. 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.563+0000 7f8b03a27640 10 monclient: build_initial_monmap 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.563+0000 7f8b03a27640 10 monclient: monmap: 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.563+0000 7f8b03a27640 1 librados: starting msgr at 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.563+0000 7f8b03a27640 1 librados: starting objecter 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.564+0000 7f8b03a27640 1 Processor -- start 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.564+0000 7f8b03a27640 1 -- start start 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.564+0000 7f8b03a27640 1 librados: setting wanted keys 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.564+0000 7f8b03a27640 1 librados: calling monclient init 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.564+0000 7f8b03a27640 10 monclient: init 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.565+0000 7f8b03a27640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.565+0000 7f8b03a27640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.565+0000 7f8b03a27640 10 monclient(hunting): picked mon.noname-b con 0x7f8afc0a9d70 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.565+0000 7f8b03a27640 10 monclient(hunting): picked mon.noname-c con 0x7f8afc13f650 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.565+0000 7f8b03a27640 10 monclient(hunting): picked mon.noname-a con 0x7f8afc1457f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.565+0000 7f8b03a27640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8afc0add00 con 0x7f8afc1457f0 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.565+0000 7f8b03a27640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8afc0af380 con 0x7f8afc0a9d70 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.565+0000 7f8b03a27640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8afc0ae190 con 0x7f8afc13f650 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.565+0000 7f8b03a27640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.565+0000 7f8b03a27640 10 monclient(hunting): authenticate will time out at 1110.014320s 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.572+0000 7f8b00fc2640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f8afc13f650 0x7f8afc139d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:46521/0 (socket says 172.21.15.37:46521) 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.572+0000 7f8b00fc2640 1 -- 172.21.15.37:0/4046193 learned_addr learned my addr 172.21.15.37:0/4046193 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.573+0000 7f8af1ffb640 10 client.?.objecter ms_handle_connect 0x7f8afc13f650 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.573+0000 7f8af1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:50.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.574+0000 7f8af1ffb640 10 client.?.objecter ms_handle_connect 0x7f8afc1457f0 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.574+0000 7f8af1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.574+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199988992 0 0) 0x7f8afc0add00 con 0x7f8afc1457f0 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.574+0000 7f8af1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.574+0000 7f8af1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.574+0000 7f8af1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.574+0000 7f8af1ffb640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.574+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ae0004eb0 con 0x7f8afc1457f0 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.574+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591746850 0 0) 0x7f8afc0ae190 con 0x7f8afc13f650 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.574+0000 7f8af1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.574+0000 7f8af1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.574+0000 7f8af1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.574+0000 7f8af1ffb640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.574+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8afc0add00 con 0x7f8afc13f650 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 594076213 0 0) 0x7f8ae0004eb0 con 0x7f8afc1457f0 2024-03-29T11:38:50.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8afc0ae190 con 0x7f8afc1457f0 2024-03-29T11:38:50.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8af4001c70 con 0x7f8afc1457f0 2024-03-29T11:38:50.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:50.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:50.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:50.580 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:50.580 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:50.580 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:50.580 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:50.580 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:50.580 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:50.580 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:50.580 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:50.580 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:50.580 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:50.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:50.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3799847687 0 0) 0x7f8afc0ae190 con 0x7f8afc1457f0 2024-03-29T11:38:50.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 >> v1:172.21.15.37:6790/0 conn(0x7f8afc13f650 legacy=0x7f8afc139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 >> v1:172.21.15.146:6789/0 conn(0x7f8afc0a9d70 legacy=0x7f8afc1d7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:50.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 1 monclient: found mon.a 2024-03-29T11:38:50.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:50.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:38:50.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8afc0ad190 con 0x7f8afc1457f0 2024-03-29T11:38:50.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:50.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:50.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.575+0000 7f8af1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.576+0000 7f8b03a27640 5 monclient: authenticate success, global_id 4365 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.576+0000 7f8b03a27640 10 monclient: _renew_subs 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.576+0000 7f8b03a27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.576+0000 7f8b03a27640 1 -- 172.21.15.37:0/4046193 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8afc0ad3e0 con 0x7f8afc1457f0 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.576+0000 7f8b03a27640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.577+0000 7f8b03a27640 10 monclient: _renew_subs 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.577+0000 7f8b03a27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.577+0000 7f8b03a27640 1 -- 172.21.15.37:0/4046193 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8afc142110 con 0x7f8afc1457f0 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.577+0000 7f8b03a27640 1 librados: init done 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.577+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8af4003990 con 0x7f8afc1457f0 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.577+0000 7f8af1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.577+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8af4003d60 con 0x7f8afc1457f0 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.577+0000 7f8af1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.577+0000 7f8af1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:50.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.577+0000 7f8af1ffb640 10 monclient: dump: 2024-03-29T11:38:50.583 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:50.583 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:50.583 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:50.583 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:50.583 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:50.583 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:50.583 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:50.583 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:50.583 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:50.583 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:50.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.578+0000 7f8af2ffd640 4 set_mon_vals no callback set 2024-03-29T11:38:50.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.578+0000 7f8b03a27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:50.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.579+0000 7f8b03a27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:50.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.579+0000 7f8b03a27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:38:50.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.579+0000 7f8b03a27640 1 -- 172.21.15.37:0/4046193 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8adc005080 con 0x7f8afc1457f0 2024-03-29T11:38:50.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.580+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f8af4003ec0 con 0x7f8afc1457f0 2024-03-29T11:38:50.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.582+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f8af4083bd0 con 0x7f8afc1457f0 2024-03-29T11:38:50.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.582+0000 7f8af1ffb640 10 client.4365.objecter ms_dispatch 0x7f8afc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:50.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.582+0000 7f8af1ffb640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:50.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.582+0000 7f8af1ffb640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:50.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.582+0000 7f8af1ffb640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-29T11:38:50.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.587+0000 7f8b017c3640 10 monclient: get_auth_request con 0x7f8ae006dd00 auth_method 0 2024-03-29T11:38:50.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.588+0000 7f8af1ffb640 10 client.4365.objecter ms_handle_connect 0x7f8ae006dd00 2024-03-29T11:38:50.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.591+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8af404f4d0 con 0x7f8afc1457f0 2024-03-29T11:38:50.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.591+0000 7f8af1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:50.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.591+0000 7f8af1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:50.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.921+0000 7f8b03a27640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:50.924 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.921+0000 7f8b03a27640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:50.924 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.921+0000 7f8b03a27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:38:50.924 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.921+0000 7f8b03a27640 1 -- 172.21.15.37:0/4046193 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8adc005330 con 0x7f8afc1457f0 2024-03-29T11:38:50.924 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.922+0000 7f8af1ffb640 1 -- 172.21.15.37:0/4046193 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f8af4051400 con 0x7f8afc1457f0 2024-03-29T11:38:50.924 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.922+0000 7f8af1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:50.924 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.922+0000 7f8af1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:50.924 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:38:50.925 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:50.926 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.926+0000 7f8b03a27640 10 librados: watch_flush enter 2024-03-29T11:38:50.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.927+0000 7f8b03a27640 10 librados: watch_flush exit 2024-03-29T11:38:50.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.927+0000 7f8b03a27640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-29T11:38:50.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.927+0000 7f8b03a27640 10 client.4365.objecter successfully canceled tick 2024-03-29T11:38:50.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.927+0000 7f8b03a27640 1 -- 172.21.15.37:0/4046193 >> v1:172.21.15.146:6816/2899149139 conn(0x7f8ae006dd00 legacy=0x7f8ae00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.927+0000 7f8b03a27640 10 monclient: shutdown 2024-03-29T11:38:50.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.927+0000 7f8b03a27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:50.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.927+0000 7f8b03a27640 1 -- 172.21.15.37:0/4046193 >> v1:172.21.15.37:6789/0 conn(0x7f8afc1457f0 legacy=0x7f8afc1e2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.927+0000 7f8b03a27640 1 -- 172.21.15.37:0/4046193 shutdown_connections 2024-03-29T11:38:50.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.927+0000 7f8b03a27640 1 -- 172.21.15.37:0/4046193 >> 172.21.15.37:0/4046193 conn(0x7f8afc066a00 msgr2=0x7f8afc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:50.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.931+0000 7f8b03a27640 1 -- 172.21.15.37:0/4046193 shutdown_connections 2024-03-29T11:38:50.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.931+0000 7f8b03a27640 1 -- 172.21.15.37:0/4046193 wait complete. 2024-03-29T11:38:50.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:50.931+0000 7f8b03a27640 1 librados: shutdown 2024-03-29T11:38:50.952 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:57.759 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:57.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.908+0000 7f25aceff640 10 monclient: get_monmap_and_config 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.908+0000 7f25aceff640 10 monclient: build_initial_monmap 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.908+0000 7f25aceff640 10 monclient: monmap: 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:38:57.910811+0000 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:38:57.910811+0000 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:57.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.912+0000 7f25aceff640 1 Processor -- start 2024-03-29T11:38:57.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.912+0000 7f25aceff640 1 -- start start 2024-03-29T11:38:57.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.912+0000 7f25aceff640 10 monclient: init 2024-03-29T11:38:57.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.914+0000 7f25aceff640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:57.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.914+0000 7f25aceff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:57.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.914+0000 7f25aceff640 10 monclient(hunting): picked mon.noname-c con 0x7f25a81457f0 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:57.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.915+0000 7f25aceff640 10 monclient(hunting): picked mon.noname-a con 0x7f25a813f650 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:57.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.915+0000 7f25aceff640 10 monclient(hunting): picked mon.noname-b con 0x7f25a80a9d70 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:57.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.915+0000 7f25aceff640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a80aaef0 con 0x7f25a813f650 2024-03-29T11:38:57.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.915+0000 7f25aceff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a8145bd0 con 0x7f25a81457f0 2024-03-29T11:38:57.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.915+0000 7f25aceff640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a8145eb0 con 0x7f25a80a9d70 2024-03-29T11:38:57.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.915+0000 7f25aceff640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:57.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.915+0000 7f25aceff640 10 monclient(hunting): authenticate will time out at 1117.365691s 2024-03-29T11:38:57.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.917+0000 7f25a659c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f25a81457f0 0x7f25a813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:52271/0 (socket says 172.21.15.37:52271) 2024-03-29T11:38:57.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.917+0000 7f25a659c640 1 -- 172.21.15.37:0/3104737876 learned_addr learned my addr 172.21.15.37:0/3104737876 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:57.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.918+0000 7f25a559a640 1 -- 172.21.15.37:0/3104737876 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050362169 0 0) 0x7f25a8145bd0 con 0x7f25a81457f0 2024-03-29T11:38:57.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.918+0000 7f25a559a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:57.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.918+0000 7f25a559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:57.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.918+0000 7f25a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:57.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.918+0000 7f25a559a640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:38:57.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.918+0000 7f25a559a640 1 -- 172.21.15.37:0/3104737876 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2590003520 con 0x7f25a81457f0 2024-03-29T11:38:57.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.918+0000 7f25a559a640 1 -- 172.21.15.37:0/3104737876 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1944815194 0 0) 0x7f2590003520 con 0x7f25a81457f0 2024-03-29T11:38:57.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.918+0000 7f25a559a640 1 -- 172.21.15.37:0/3104737876 >> v1:172.21.15.37:6790/0 conn(0x7f25a80a9d70 legacy=0x7f25a814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:57.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.919+0000 7f25a559a640 1 -- 172.21.15.37:0/3104737876 >> v1:172.21.15.37:6789/0 conn(0x7f25a813f650 legacy=0x7f25a80b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:57.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.919+0000 7f25a559a640 10 monclient: _finish_hunting 0 2024-03-29T11:38:57.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.919+0000 7f25a559a640 1 monclient: found mon.noname-c 2024-03-29T11:38:57.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.919+0000 7f25a559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.919+0000 7f25a559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.919+0000 7f25a559a640 1 -- 172.21.15.37:0/3104737876 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a8146340 con 0x7f25a81457f0 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.919+0000 7f25a559a640 10 monclient: _finish_auth 0 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.919+0000 7f25a559a640 10 monclient: _check_auth_tickets 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.919+0000 7f25a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.919+0000 7f25a559a640 1 -- 172.21.15.37:0/3104737876 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f259c001690 con 0x7f25a81457f0 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.919+0000 7f25a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.919+0000 7f25a559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.919+0000 7f25a559a640 10 monclient: dump: 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.921+0000 7f25aceff640 5 monclient: authenticate success, global_id 4348 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.921+0000 7f25aceff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:57.925 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.923+0000 7f25a559a640 1 -- 172.21.15.37:0/3104737876 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f259c001860 con 0x7f25a81457f0 2024-03-29T11:38:57.926 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.923+0000 7f25a559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.923+0000 7f25a559a640 1 -- 172.21.15.37:0/3104737876 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f259c001cb0 con 0x7f25a81457f0 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.923+0000 7f25a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.923+0000 7f25a559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.923+0000 7f25a559a640 10 monclient: dump: 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.924+0000 7f25aceff640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.924+0000 7f25aceff640 4 set_mon_vals no callback set 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.925+0000 7f25aceff640 10 monclient: shutdown 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.925+0000 7f25aceff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.925+0000 7f25aceff640 1 -- 172.21.15.37:0/3104737876 >> v1:172.21.15.146:6789/0 conn(0x7f25a81457f0 legacy=0x7f25a813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.926+0000 7f25aceff640 1 -- 172.21.15.37:0/3104737876 shutdown_connections 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.926+0000 7f25aceff640 1 -- 172.21.15.37:0/3104737876 >> 172.21.15.37:0/3104737876 conn(0x7f25a8066a00 msgr2=0x7f25a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.926+0000 7f25aceff640 1 -- 172.21.15.37:0/3104737876 shutdown_connections 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.927+0000 7f25aceff640 1 -- 172.21.15.37:0/3104737876 wait complete. 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.928+0000 7f25aceff640 10 monclient: build_initial_monmap 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.928+0000 7f25aceff640 10 monclient: monmap: 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:38:57.932 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:57.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.928+0000 7f25aceff640 1 librados: starting msgr at 2024-03-29T11:38:57.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.928+0000 7f25aceff640 1 librados: starting objecter 2024-03-29T11:38:57.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.928+0000 7f25aceff640 1 Processor -- start 2024-03-29T11:38:57.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.932+0000 7f25aceff640 1 -- start start 2024-03-29T11:38:57.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.932+0000 7f25aceff640 1 librados: setting wanted keys 2024-03-29T11:38:57.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.932+0000 7f25aceff640 1 librados: calling monclient init 2024-03-29T11:38:57.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.932+0000 7f25aceff640 10 monclient: init 2024-03-29T11:38:57.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.933+0000 7f25aceff640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:57.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.934+0000 7f25aceff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:57.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.934+0000 7f25aceff640 10 monclient(hunting): picked mon.noname-c con 0x7f25a80a9d70 addr v1:172.21.15.37:6790/0 2024-03-29T11:38:57.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.937+0000 7f25aceff640 10 monclient(hunting): picked mon.noname-b con 0x7f25a813f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:38:57.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.937+0000 7f25aceff640 10 monclient(hunting): picked mon.noname-a con 0x7f25a81457f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:38:57.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.937+0000 7f25aceff640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a81352a0 con 0x7f25a81457f0 2024-03-29T11:38:57.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.937+0000 7f25aceff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a8133e40 con 0x7f25a813f650 2024-03-29T11:38:57.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.937+0000 7f25aceff640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a80b1730 con 0x7f25a80a9d70 2024-03-29T11:38:57.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.937+0000 7f25aceff640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:57.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.937+0000 7f25aceff640 10 monclient(hunting): authenticate will time out at 1117.387901s 2024-03-29T11:38:57.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.938+0000 7f25a659c640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f25a80a9d70 0x7f25a812fdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:43267/0 (socket says 172.21.15.37:43267) 2024-03-29T11:38:57.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.938+0000 7f25a659c640 1 -- 172.21.15.37:0/3848340718 learned_addr learned my addr 172.21.15.37:0/3848340718 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:38:57.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.938+0000 7f2596ffd640 10 client.?.objecter ms_handle_connect 0x7f25a80a9d70 2024-03-29T11:38:57.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.938+0000 7f2596ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:57.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.938+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736947494 0 0) 0x7f25a80b1730 con 0x7f25a80a9d70 2024-03-29T11:38:57.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.938+0000 7f2596ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:57.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.938+0000 7f2596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:57.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.938+0000 7f2596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:57.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.938+0000 7f2596ffd640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:38:57.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.938+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2588004eb0 con 0x7f25a80a9d70 2024-03-29T11:38:57.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.944+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1580900810 0 0) 0x7f2588004eb0 con 0x7f25a80a9d70 2024-03-29T11:38:57.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.944+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25a80b1730 con 0x7f25a80a9d70 2024-03-29T11:38:57.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.944+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f259c001c30 con 0x7f25a80a9d70 2024-03-29T11:38:57.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.944+0000 7f2596ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:57.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.944+0000 7f2596ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:57.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.944+0000 7f2596ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:57.948 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:57.948 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:57.948 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:57.948 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:57.948 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:57.948 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:57.948 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:57.948 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.944+0000 7f2596ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.944+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1214298712 0 0) 0x7f25a80b1730 con 0x7f25a80a9d70 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.944+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 >> v1:172.21.15.146:6789/0 conn(0x7f25a813f650 legacy=0x7f25a81359a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.944+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 >> v1:172.21.15.37:6789/0 conn(0x7f25a81457f0 legacy=0x7f25a81360b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f2596ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f2596ffd640 1 monclient: found mon.c 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f2596ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f2596ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a80b18c0 con 0x7f25a80a9d70 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f2596ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f2596ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f2596ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f25aceff640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:38:57.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f25aceff640 10 monclient: _renew_subs 2024-03-29T11:38:57.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f25aceff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:57.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f25aceff640 1 -- 172.21.15.37:0/3848340718 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a81304d0 con 0x7f25a80a9d70 2024-03-29T11:38:57.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f25aceff640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:57.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f25aceff640 10 monclient: _renew_subs 2024-03-29T11:38:57.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f25aceff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:57.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f25aceff640 1 -- 172.21.15.37:0/3848340718 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a8141aa0 con 0x7f25a80a9d70 2024-03-29T11:38:57.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f25aceff640 1 librados: init done 2024-03-29T11:38:57.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f259c004a90 con 0x7f25a80a9d70 2024-03-29T11:38:57.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f2596ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:57.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f259c004d40 con 0x7f25a80a9d70 2024-03-29T11:38:57.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f2596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:57.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f2596ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:57.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f2596ffd640 10 monclient: dump: 2024-03-29T11:38:57.950 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:38:57.951 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:38:57.951 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:57.951 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:38:57.951 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:57.951 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:38:57.951 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:38:57.951 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:38:57.951 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:38:57.951 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:38:57.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.945+0000 7f25a4d99640 4 set_mon_vals no callback set 2024-03-29T11:38:57.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.947+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f259c004ea0 con 0x7f25a80a9d70 2024-03-29T11:38:57.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.948+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f259c084a50 con 0x7f25a80a9d70 2024-03-29T11:38:57.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.948+0000 7f2596ffd640 10 client.4385.objecter ms_dispatch 0x7f25a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:57.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.948+0000 7f25a5d9b640 10 monclient: get_auth_request con 0x7f258806dc60 auth_method 0 2024-03-29T11:38:57.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.948+0000 7f2596ffd640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:57.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.948+0000 7f2596ffd640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:57.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.948+0000 7f2596ffd640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T11:38:57.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.948+0000 7f2596ffd640 10 client.4385.objecter ms_handle_connect 0x7f258806dc60 2024-03-29T11:38:57.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.948+0000 7f25aceff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:57.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.948+0000 7f25aceff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:57.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.948+0000 7f25aceff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:57.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.949+0000 7f25aceff640 1 -- 172.21.15.37:0/3848340718 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f258c005080 con 0x7f25a80a9d70 2024-03-29T11:38:57.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.959+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f259c050330 con 0x7f25a80a9d70 2024-03-29T11:38:57.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.959+0000 7f2596ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:57.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:57.959+0000 7f2596ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:58.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.288+0000 7f25aceff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:58.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.288+0000 7f25aceff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:58.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.288+0000 7f25aceff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:38:58.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.288+0000 7f25aceff640 1 -- 172.21.15.37:0/3848340718 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f258c005330 con 0x7f25a80a9d70 2024-03-29T11:38:58.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.288+0000 7f2596ffd640 1 -- 172.21.15.37:0/3848340718 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f259c052260 con 0x7f25a80a9d70 2024-03-29T11:38:58.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.288+0000 7f2596ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:58.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.288+0000 7f2596ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:58.291 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:38:58.291 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:58.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.292+0000 7f25aceff640 10 librados: watch_flush enter 2024-03-29T11:38:58.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.292+0000 7f25aceff640 10 librados: watch_flush exit 2024-03-29T11:38:58.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.292+0000 7f25aceff640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T11:38:58.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.292+0000 7f25aceff640 10 client.4385.objecter successfully canceled tick 2024-03-29T11:38:58.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.292+0000 7f25aceff640 1 -- 172.21.15.37:0/3848340718 >> v1:172.21.15.146:6816/2899149139 conn(0x7f258806dc60 legacy=0x7f2588070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:58.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.292+0000 7f25aceff640 10 monclient: shutdown 2024-03-29T11:38:58.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.292+0000 7f25aceff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:58.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.292+0000 7f25aceff640 1 -- 172.21.15.37:0/3848340718 >> v1:172.21.15.37:6790/0 conn(0x7f25a80a9d70 legacy=0x7f25a812fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:58.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.292+0000 7f25aceff640 1 -- 172.21.15.37:0/3848340718 shutdown_connections 2024-03-29T11:38:58.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.292+0000 7f25aceff640 1 -- 172.21.15.37:0/3848340718 >> 172.21.15.37:0/3848340718 conn(0x7f25a8066a00 msgr2=0x7f25a80b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:58.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.293+0000 7f25aceff640 1 -- 172.21.15.37:0/3848340718 shutdown_connections 2024-03-29T11:38:58.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.293+0000 7f25aceff640 1 -- 172.21.15.37:0/3848340718 wait complete. 2024-03-29T11:38:58.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:38:58.294+0000 7f25aceff640 1 librados: shutdown 2024-03-29T11:38:58.315 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:05.121 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:05.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.277+0000 7f8b0c643640 10 monclient: get_monmap_and_config 2024-03-29T11:39:05.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.277+0000 7f8b0c643640 10 monclient: build_initial_monmap 2024-03-29T11:39:05.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.277+0000 7f8b0c643640 10 monclient: monmap: 2024-03-29T11:39:05.278 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:05.278 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:05.278 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:39:05.278081+0000 2024-03-29T11:39:05.278 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:39:05.278081+0000 2024-03-29T11:39:05.278 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:05.278 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:05.278 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:05.278 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:39:05.279 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:39:05.279 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:05.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.280+0000 7f8b0c643640 1 Processor -- start 2024-03-29T11:39:05.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.281+0000 7f8b0c643640 1 -- start start 2024-03-29T11:39:05.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.281+0000 7f8b0c643640 10 monclient: init 2024-03-29T11:39:05.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.282+0000 7f8b0c643640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:05.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.283+0000 7f8b0c643640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:05.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.283+0000 7f8b0c643640 10 monclient(hunting): picked mon.noname-c con 0x7f8b040b1f60 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:05.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.283+0000 7f8b0c643640 10 monclient(hunting): picked mon.noname-b con 0x7f8b04071eb0 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:05.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.283+0000 7f8b0c643640 10 monclient(hunting): picked mon.noname-a con 0x7f8b04143d10 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:05.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.283+0000 7f8b0c643640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0406ef80 con 0x7f8b04143d10 2024-03-29T11:39:05.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.283+0000 7f8b0c643640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0406f260 con 0x7f8b040b1f60 2024-03-29T11:39:05.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.283+0000 7f8b0c643640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b040b40c0 con 0x7f8b04071eb0 2024-03-29T11:39:05.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.283+0000 7f8b0c643640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:05.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.283+0000 7f8b0c643640 10 monclient(hunting): authenticate will time out at 1124.732781s 2024-03-29T11:39:05.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.287+0000 7f8b09bde640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f8b04071eb0 0x7f8b04146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:45137/0 (socket says 172.21.15.37:45137) 2024-03-29T11:39:05.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.287+0000 7f8b09bde640 1 -- 172.21.15.37:0/903835367 learned_addr learned my addr 172.21.15.37:0/903835367 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:05.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 1 -- 172.21.15.37:0/903835367 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639489859 0 0) 0x7f8b040b40c0 con 0x7f8b04071eb0 2024-03-29T11:39:05.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:05.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:05.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:05.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:39:05.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 1 -- 172.21.15.37:0/903835367 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af0003520 con 0x7f8b04071eb0 2024-03-29T11:39:05.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 1 -- 172.21.15.37:0/903835367 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069221512 0 0) 0x7f8b0406f260 con 0x7f8b040b1f60 2024-03-29T11:39:05.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:05.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:05.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:05.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:39:05.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 1 -- 172.21.15.37:0/903835367 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b040b40c0 con 0x7f8b040b1f60 2024-03-29T11:39:05.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 1 -- 172.21.15.37:0/903835367 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2661048942 0 0) 0x7f8af0003520 con 0x7f8b04071eb0 2024-03-29T11:39:05.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 1 -- 172.21.15.37:0/903835367 >> v1:172.21.15.146:6789/0 conn(0x7f8b040b1f60 legacy=0x7f8b04143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.291+0000 7f8b093dd640 1 -- 172.21.15.37:0/903835367 >> v1:172.21.15.37:6789/0 conn(0x7f8b04143d10 legacy=0x7f8b04146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:05.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.293+0000 7f8b093dd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:05.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.293+0000 7f8b093dd640 1 monclient: found mon.noname-b 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.293+0000 7f8b093dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.293+0000 7f8b093dd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.293+0000 7f8b093dd640 1 -- 172.21.15.37:0/903835367 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b04155d70 con 0x7f8b04071eb0 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.293+0000 7f8b093dd640 10 monclient: _finish_auth 0 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.293+0000 7f8b093dd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.293+0000 7f8b093dd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.293+0000 7f8b093dd640 1 -- 172.21.15.37:0/903835367 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8af40015c0 con 0x7f8b04071eb0 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.293+0000 7f8b093dd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.293+0000 7f8b093dd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.293+0000 7f8b093dd640 10 monclient: dump: 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:05.295 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.296 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:05.296 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:05.296 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:05.296 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:05.296 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:05.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.294+0000 7f8b0c643640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:39:05.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.294+0000 7f8b0c643640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:05.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.295+0000 7f8b093dd640 1 -- 172.21.15.37:0/903835367 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8af4001720 con 0x7f8b04071eb0 2024-03-29T11:39:05.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.295+0000 7f8b093dd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:05.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.295+0000 7f8b093dd640 1 -- 172.21.15.37:0/903835367 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8af4001b30 con 0x7f8b04071eb0 2024-03-29T11:39:05.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.295+0000 7f8b0c643640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:05.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.295+0000 7f8b0c643640 4 set_mon_vals no callback set 2024-03-29T11:39:05.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.296+0000 7f8b093dd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.296+0000 7f8b093dd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.296+0000 7f8b093dd640 10 monclient: dump: 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.296+0000 7f8b0c643640 10 monclient: shutdown 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.296+0000 7f8b0c643640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:05.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.296+0000 7f8b0c643640 1 -- 172.21.15.37:0/903835367 >> v1:172.21.15.37:6790/0 conn(0x7f8b04071eb0 legacy=0x7f8b04146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.299+0000 7f8b0c643640 1 -- 172.21.15.37:0/903835367 shutdown_connections 2024-03-29T11:39:05.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.299+0000 7f8b0c643640 1 -- 172.21.15.37:0/903835367 >> 172.21.15.37:0/903835367 conn(0x7f8b04066a00 msgr2=0x7f8b04066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:05.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.301+0000 7f8b0c643640 1 -- 172.21.15.37:0/903835367 shutdown_connections 2024-03-29T11:39:05.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.301+0000 7f8b0c643640 1 -- 172.21.15.37:0/903835367 wait complete. 2024-03-29T11:39:05.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.302+0000 7f8b0c643640 10 monclient: build_initial_monmap 2024-03-29T11:39:05.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.303+0000 7f8b0c643640 10 monclient: monmap: 2024-03-29T11:39:05.303 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:05.303 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:05.303 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:39:05.303 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:39:05.303 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:05.304 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:05.304 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:05.304 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:39:05.304 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:39:05.304 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:05.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.304+0000 7f8b0c643640 1 librados: starting msgr at 2024-03-29T11:39:05.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.304+0000 7f8b0c643640 1 librados: starting objecter 2024-03-29T11:39:05.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.304+0000 7f8b0c643640 1 Processor -- start 2024-03-29T11:39:05.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.305+0000 7f8b0c643640 1 -- start start 2024-03-29T11:39:05.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.305+0000 7f8b0c643640 1 librados: setting wanted keys 2024-03-29T11:39:05.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.305+0000 7f8b0c643640 1 librados: calling monclient init 2024-03-29T11:39:05.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.305+0000 7f8b0c643640 10 monclient: init 2024-03-29T11:39:05.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.306+0000 7f8b0c643640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:05.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.310+0000 7f8b0c643640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:05.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.311+0000 7f8b0c643640 10 monclient(hunting): picked mon.noname-a con 0x7f8b040b1f60 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:05.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.311+0000 7f8b0c643640 10 monclient(hunting): picked mon.noname-c con 0x7f8b04143d10 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:05.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.311+0000 7f8b0c643640 10 monclient(hunting): picked mon.noname-b con 0x7f8b04154720 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:05.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.311+0000 7f8b0c643640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b041497e0 con 0x7f8b040b1f60 2024-03-29T11:39:05.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.311+0000 7f8b0c643640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b04147d40 con 0x7f8b04154720 2024-03-29T11:39:05.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.311+0000 7f8b0c643640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0413b8d0 con 0x7f8b04143d10 2024-03-29T11:39:05.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.311+0000 7f8b0c643640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:05.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.311+0000 7f8b0c643640 10 monclient(hunting): authenticate will time out at 1124.760026s 2024-03-29T11:39:05.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.314+0000 7f8b09bde640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f8b04143d10 0x7f8b0413aab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:45279/0 (socket says 172.21.15.37:45279) 2024-03-29T11:39:05.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.314+0000 7f8b0a3df640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8b040b1f60 0x7f8b04147630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:44181/0 (socket says 172.21.15.37:44181) 2024-03-29T11:39:05.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.314+0000 7f8b0a3df640 1 -- 172.21.15.37:0/342886931 learned_addr learned my addr 172.21.15.37:0/342886931 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:05.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.314+0000 7f8afaffd640 10 client.?.objecter ms_handle_connect 0x7f8b040b1f60 2024-03-29T11:39:05.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.314+0000 7f8afaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.314+0000 7f8afaffd640 10 client.?.objecter ms_handle_connect 0x7f8b04143d10 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.314+0000 7f8afaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.314+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807485589 0 0) 0x7f8b0413b8d0 con 0x7f8b04143d10 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.315+0000 7f8afaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.315+0000 7f8afaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.315+0000 7f8afaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.315+0000 7f8afaffd640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.315+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ae8004eb0 con 0x7f8b04143d10 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.315+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037458161 0 0) 0x7f8b041497e0 con 0x7f8b040b1f60 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.315+0000 7f8afaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.315+0000 7f8afaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.315+0000 7f8afaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.315+0000 7f8afaffd640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.315+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b0413b8d0 con 0x7f8b040b1f60 2024-03-29T11:39:05.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 987106084 0 0) 0x7f8b0413b8d0 con 0x7f8b040b1f60 2024-03-29T11:39:05.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b041497e0 con 0x7f8b040b1f60 2024-03-29T11:39:05.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8b00001970 con 0x7f8b040b1f60 2024-03-29T11:39:05.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:05.322 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 10 monclient(hunting): dump: 2024-03-29T11:39:05.322 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:05.322 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:05.322 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:05.322 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:05.322 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.322 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:05.322 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:05.322 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:05.322 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:05.322 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:05.322 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 281849250 0 0) 0x7f8b041497e0 con 0x7f8b040b1f60 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 >> v1:172.21.15.37:6790/0 conn(0x7f8b04143d10 legacy=0x7f8b0413aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 >> v1:172.21.15.146:6789/0 conn(0x7f8b04154720 legacy=0x7f8b0413b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 1 monclient: found mon.a 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0414ae50 con 0x7f8b040b1f60 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8afaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8b0c643640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8b0c643640 10 monclient: _renew_subs 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8b0c643640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8b0c643640 1 -- 172.21.15.37:0/342886931 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0414b130 con 0x7f8b040b1f60 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8b0c643640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.318+0000 7f8b0c643640 10 monclient: _renew_subs 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.319+0000 7f8b0c643640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.319+0000 7f8b0c643640 1 -- 172.21.15.37:0/342886931 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b04148f40 con 0x7f8b040b1f60 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.319+0000 7f8b0c643640 1 librados: init done 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.319+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b00003610 con 0x7f8b040b1f60 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.319+0000 7f8afaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.319+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8b00003a20 con 0x7f8b040b1f60 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.319+0000 7f8afaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.319+0000 7f8afaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.320+0000 7f8afaffd640 10 monclient: dump: 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.320+0000 7f8b0c643640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.320+0000 7f8b0c643640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.320+0000 7f8b0c643640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.320+0000 7f8afbfff640 4 set_mon_vals no callback set 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.320+0000 7f8b0c643640 1 -- 172.21.15.37:0/342886931 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b04149550 con 0x7f8b040b1f60 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.322+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f8b00003b80 con 0x7f8b040b1f60 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.323+0000 7f8b09bde640 10 monclient: get_auth_request con 0x7f8ae806db40 auth_method 0 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.323+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f8b00084a40 con 0x7f8b040b1f60 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.323+0000 7f8afaffd640 10 client.4368.objecter ms_dispatch 0x7f8b04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.323+0000 7f8afaffd640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.324+0000 7f8afaffd640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.324+0000 7f8afaffd640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.324+0000 7f8afaffd640 10 client.4368.objecter ms_handle_connect 0x7f8ae806db40 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.332+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b00050150 con 0x7f8b040b1f60 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.332+0000 7f8afaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.332+0000 7f8afaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:05.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.625+0000 7f8b0c643640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:05.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.625+0000 7f8b0c643640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:05.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.625+0000 7f8b0c643640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:39:05.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.625+0000 7f8b0c643640 1 -- 172.21.15.37:0/342886931 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b04075f10 con 0x7f8b040b1f60 2024-03-29T11:39:05.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.626+0000 7f8afaffd640 1 -- 172.21.15.37:0/342886931 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f8b00052250 con 0x7f8b040b1f60 2024-03-29T11:39:05.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.626+0000 7f8afaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:05.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.626+0000 7f8afaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:05.627 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:39:05.627 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:05.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.629+0000 7f8b0c643640 10 librados: watch_flush enter 2024-03-29T11:39:05.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.632+0000 7f8b0c643640 10 librados: watch_flush exit 2024-03-29T11:39:05.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.632+0000 7f8b0c643640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-29T11:39:05.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.632+0000 7f8b0c643640 10 client.4368.objecter successfully canceled tick 2024-03-29T11:39:05.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.632+0000 7f8b0c643640 1 -- 172.21.15.37:0/342886931 >> v1:172.21.15.146:6816/2899149139 conn(0x7f8ae806db40 legacy=0x7f8ae8070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.632+0000 7f8b0c643640 10 monclient: shutdown 2024-03-29T11:39:05.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.632+0000 7f8b0c643640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:05.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.632+0000 7f8b0c643640 1 -- 172.21.15.37:0/342886931 >> v1:172.21.15.37:6789/0 conn(0x7f8b040b1f60 legacy=0x7f8b04147630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.632+0000 7f8b0c643640 1 -- 172.21.15.37:0/342886931 shutdown_connections 2024-03-29T11:39:05.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.632+0000 7f8b0c643640 1 -- 172.21.15.37:0/342886931 >> 172.21.15.37:0/342886931 conn(0x7f8b04066a00 msgr2=0x7f8b040b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:05.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.634+0000 7f8b0c643640 1 -- 172.21.15.37:0/342886931 shutdown_connections 2024-03-29T11:39:05.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.634+0000 7f8b0c643640 1 -- 172.21.15.37:0/342886931 wait complete. 2024-03-29T11:39:05.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:05.634+0000 7f8b0c643640 1 librados: shutdown 2024-03-29T11:39:05.652 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:12.459 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:12.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.607+0000 7f2f0cc63640 10 monclient: get_monmap_and_config 2024-03-29T11:39:12.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.607+0000 7f2f0cc63640 10 monclient: build_initial_monmap 2024-03-29T11:39:12.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.607+0000 7f2f0cc63640 10 monclient: monmap: 2024-03-29T11:39:12.609 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:12.609 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:12.609 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:39:12.608270+0000 2024-03-29T11:39:12.609 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:39:12.608270+0000 2024-03-29T11:39:12.609 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:12.609 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:12.609 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:12.610 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:39:12.610 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:39:12.610 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:12.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.610+0000 7f2f0cc63640 1 Processor -- start 2024-03-29T11:39:12.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.611+0000 7f2f0cc63640 1 -- start start 2024-03-29T11:39:12.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.611+0000 7f2f0cc63640 10 monclient: init 2024-03-29T11:39:12.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.612+0000 7f2f0cc63640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:12.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.612+0000 7f2f0cc63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:12.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.612+0000 7f2f0cc63640 10 monclient(hunting): picked mon.noname-c con 0x7f2f080a9ac0 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:12.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.612+0000 7f2f0cc63640 10 monclient(hunting): picked mon.noname-a con 0x7f2f0813ee90 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:12.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.612+0000 7f2f0cc63640 10 monclient(hunting): picked mon.noname-b con 0x7f2f080af900 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.612+0000 7f2f0cc63640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f080afd10 con 0x7f2f0813ee90 2024-03-29T11:39:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.612+0000 7f2f0cc63640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f080af6f0 con 0x7f2f080a9ac0 2024-03-29T11:39:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.612+0000 7f2f0cc63640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f080aac80 con 0x7f2f080af900 2024-03-29T11:39:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.612+0000 7f2f0cc63640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.612+0000 7f2f0cc63640 10 monclient(hunting): authenticate will time out at 1132.061660s 2024-03-29T11:39:12.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.616+0000 7f2f05d9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f2f0813ee90 0x7f2f0813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52187/0 (socket says 172.21.15.37:52187) 2024-03-29T11:39:12.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.620+0000 7f2f0659c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2f080a9ac0 0x7f2f080adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:39469/0 (socket says 172.21.15.37:39469) 2024-03-29T11:39:12.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.620+0000 7f2f05d9b640 1 -- 172.21.15.37:0/876841718 learned_addr learned my addr 172.21.15.37:0/876841718 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:12.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.617+0000 7f2f06d9d640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f2f080af900 0x7f2f08149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:38007/0 (socket says 172.21.15.37:38007) 2024-03-29T11:39:12.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.621+0000 7f2f0559a640 1 -- 172.21.15.37:0/876841718 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967803034 0 0) 0x7f2f080aac80 con 0x7f2f080af900 2024-03-29T11:39:12.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:12.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:12.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:12.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:39:12.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 1 -- 172.21.15.37:0/876841718 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef0003440 con 0x7f2f080af900 2024-03-29T11:39:12.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 1 -- 172.21.15.37:0/876841718 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1850526097 0 0) 0x7f2ef0003440 con 0x7f2f080af900 2024-03-29T11:39:12.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 1 -- 172.21.15.37:0/876841718 >> v1:172.21.15.146:6789/0 conn(0x7f2f080a9ac0 legacy=0x7f2f080adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:12.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 1 -- 172.21.15.37:0/876841718 >> v1:172.21.15.37:6789/0 conn(0x7f2f0813ee90 legacy=0x7f2f0813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:12.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 10 monclient: _finish_hunting 0 2024-03-29T11:39:12.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 1 monclient: found mon.noname-b 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 1 -- 172.21.15.37:0/876841718 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f080aaf60 con 0x7f2f080af900 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 10 monclient: _finish_auth 0 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 10 monclient: _check_auth_tickets 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 1 -- 172.21.15.37:0/876841718 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2f00001a10 con 0x7f2f080af900 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.622+0000 7f2f0559a640 10 monclient: dump: 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:12.624 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.625 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:12.625 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:12.625 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:12.625 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:12.625 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:12.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.623+0000 7f2f0559a640 1 -- 172.21.15.37:0/876841718 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f00001ca0 con 0x7f2f080af900 2024-03-29T11:39:12.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.623+0000 7f2f0559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:12.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.623+0000 7f2f0559a640 1 -- 172.21.15.37:0/876841718 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2f00002030 con 0x7f2f080af900 2024-03-29T11:39:12.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.623+0000 7f2f0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.623+0000 7f2f0559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:12.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.623+0000 7f2f0559a640 10 monclient: dump: 2024-03-29T11:39:12.626 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:12.626 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:12.626 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:12.626 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:12.626 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.626 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:12.626 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:12.626 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:12.627 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:12.627 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:12.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.623+0000 7f2f0cc63640 5 monclient: authenticate success, global_id 4394 2024-03-29T11:39:12.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.623+0000 7f2f0cc63640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:12.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.623+0000 7f2f0cc63640 4 set_mon_vals no callback set 2024-03-29T11:39:12.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.624+0000 7f2f0cc63640 10 monclient: shutdown 2024-03-29T11:39:12.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.624+0000 7f2f0cc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:12.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.624+0000 7f2f0cc63640 1 -- 172.21.15.37:0/876841718 >> v1:172.21.15.37:6790/0 conn(0x7f2f080af900 legacy=0x7f2f08149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:12.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.624+0000 7f2f0cc63640 1 -- 172.21.15.37:0/876841718 shutdown_connections 2024-03-29T11:39:12.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.624+0000 7f2f0cc63640 1 -- 172.21.15.37:0/876841718 >> 172.21.15.37:0/876841718 conn(0x7f2f080b35e0 msgr2=0x7f2f080b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:12.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.624+0000 7f2f0cc63640 1 -- 172.21.15.37:0/876841718 shutdown_connections 2024-03-29T11:39:12.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.625+0000 7f2f0cc63640 1 -- 172.21.15.37:0/876841718 wait complete. 2024-03-29T11:39:12.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.626+0000 7f2f0cc63640 10 monclient: build_initial_monmap 2024-03-29T11:39:12.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.626+0000 7f2f0cc63640 10 monclient: monmap: 2024-03-29T11:39:12.628 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:12.628 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:12.628 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:39:12.628 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:39:12.629 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:12.629 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:12.629 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:12.629 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:39:12.629 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:39:12.629 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:12.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.627+0000 7f2f0cc63640 1 librados: starting msgr at 2024-03-29T11:39:12.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.627+0000 7f2f0cc63640 1 librados: starting objecter 2024-03-29T11:39:12.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.627+0000 7f2f0cc63640 1 Processor -- start 2024-03-29T11:39:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.627+0000 7f2f0cc63640 1 -- start start 2024-03-29T11:39:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.627+0000 7f2f0cc63640 1 librados: setting wanted keys 2024-03-29T11:39:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.627+0000 7f2f0cc63640 1 librados: calling monclient init 2024-03-29T11:39:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.627+0000 7f2f0cc63640 10 monclient: init 2024-03-29T11:39:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.628+0000 7f2f0cc63640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.628+0000 7f2f0cc63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.628+0000 7f2f0cc63640 10 monclient(hunting): picked mon.noname-c con 0x7f2f080a9ac0 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.628+0000 7f2f0cc63640 10 monclient(hunting): picked mon.noname-a con 0x7f2f080af900 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.628+0000 7f2f0cc63640 10 monclient(hunting): picked mon.noname-b con 0x7f2f0813ee90 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.628+0000 7f2f0cc63640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f081425e0 con 0x7f2f080af900 2024-03-29T11:39:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.628+0000 7f2f0cc63640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f080704d0 con 0x7f2f0813ee90 2024-03-29T11:39:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.628+0000 7f2f0cc63640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f080707b0 con 0x7f2f080a9ac0 2024-03-29T11:39:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.628+0000 7f2f0cc63640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.628+0000 7f2f0cc63640 10 monclient(hunting): authenticate will time out at 1132.077615s 2024-03-29T11:39:12.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.633+0000 7f2f06d9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2f0813ee90 0x7f2f081ee7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:39549/0 (socket says 172.21.15.37:39549) 2024-03-29T11:39:12.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.633+0000 7f2f06d9d640 1 -- 172.21.15.37:0/3909765109 learned_addr learned my addr 172.21.15.37:0/3909765109 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:12.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.633+0000 7f2ef6ffd640 10 client.?.objecter ms_handle_connect 0x7f2f0813ee90 2024-03-29T11:39:12.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.633+0000 7f2ef6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:12.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.633+0000 7f2ef6ffd640 10 client.?.objecter ms_handle_connect 0x7f2f080a9ac0 2024-03-29T11:39:12.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.633+0000 7f2ef6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:12.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.634+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958145768 0 0) 0x7f2f080704d0 con 0x7f2f0813ee90 2024-03-29T11:39:12.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.634+0000 7f2ef6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:12.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.634+0000 7f2ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:12.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.634+0000 7f2ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:12.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.634+0000 7f2ef6ffd640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:39:12.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.634+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee8004eb0 con 0x7f2f0813ee90 2024-03-29T11:39:12.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.634+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967445242 0 0) 0x7f2f080707b0 con 0x7f2f080a9ac0 2024-03-29T11:39:12.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.634+0000 7f2ef6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:12.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.634+0000 7f2ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:12.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.634+0000 7f2ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:12.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.634+0000 7f2ef6ffd640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:39:12.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.634+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f080704d0 con 0x7f2f080a9ac0 2024-03-29T11:39:12.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.637+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 358384679 0 0) 0x7f2f080704d0 con 0x7f2f080a9ac0 2024-03-29T11:39:12.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.638+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f080707b0 con 0x7f2f080a9ac0 2024-03-29T11:39:12.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.638+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3944922950 0 0) 0x7f2ee8004eb0 con 0x7f2f0813ee90 2024-03-29T11:39:12.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.638+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ee8006900 con 0x7f2f0813ee90 2024-03-29T11:39:12.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.638+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2f00001990 con 0x7f2f0813ee90 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.638+0000 7f2ef6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.638+0000 7f2ef6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.638+0000 7f2ef6ffd640 10 monclient(hunting): dump: 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.638+0000 7f2ef6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.638+0000 7f2ef6ffd640 10 client.?.objecter ms_handle_connect 0x7f2f080af900 2024-03-29T11:39:12.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.638+0000 7f2ef6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.638+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207635866 0 0) 0x7f2f081425e0 con 0x7f2f080af900 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.638+0000 7f2ef6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.638+0000 7f2ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.638+0000 7f2ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee8006e00 con 0x7f2f080af900 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2ef8001510 con 0x7f2f080a9ac0 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 10 monclient(hunting): dump: 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:12.642 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2980670886 0 0) 0x7f2ee8006900 con 0x7f2f0813ee90 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 >> v1:172.21.15.37:6790/0 conn(0x7f2f080a9ac0 legacy=0x7f2f0806ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 >> v1:172.21.15.37:6789/0 conn(0x7f2f080af900 legacy=0x7f2f081e3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 1 monclient: found mon.b 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0806f650 con 0x7f2f0813ee90 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:12.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2ef6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:12.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2f0cc63640 5 monclient: authenticate success, global_id 4354 2024-03-29T11:39:12.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2f0cc63640 10 monclient: _renew_subs 2024-03-29T11:39:12.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2f0cc63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:39:12.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.639+0000 7f2f0cc63640 1 -- 172.21.15.37:0/3909765109 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0806f120 con 0x7f2f0813ee90 2024-03-29T11:39:12.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.640+0000 7f2f0cc63640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:12.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.640+0000 7f2f0cc63640 10 monclient: _renew_subs 2024-03-29T11:39:12.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.640+0000 7f2f0cc63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:39:12.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.640+0000 7f2f0cc63640 1 -- 172.21.15.37:0/3909765109 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f081e4650 con 0x7f2f0813ee90 2024-03-29T11:39:12.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.640+0000 7f2f0cc63640 1 librados: init done 2024-03-29T11:39:12.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.640+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f000015f0 con 0x7f2f0813ee90 2024-03-29T11:39:12.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.640+0000 7f2ef6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:12.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.640+0000 7f2f04d99640 4 set_mon_vals no callback set 2024-03-29T11:39:12.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.641+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2f00001990 con 0x7f2f0813ee90 2024-03-29T11:39:12.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.642+0000 7f2ef6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.642+0000 7f2ef6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:12.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.642+0000 7f2ef6ffd640 10 monclient: dump: 2024-03-29T11:39:12.646 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:12.646 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:12.646 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:12.646 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:12.646 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.646 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:12.647 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:12.647 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:12.647 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:12.647 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:12.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.642+0000 7f2f0cc63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:12.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.643+0000 7f2f0cc63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:12.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.643+0000 7f2f0cc63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:39:12.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.643+0000 7f2f0cc63640 1 -- 172.21.15.37:0/3909765109 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eec005080 con 0x7f2f0813ee90 2024-03-29T11:39:12.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.644+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f2f0001ada0 con 0x7f2f0813ee90 2024-03-29T11:39:12.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.645+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f2f000547e0 con 0x7f2f0813ee90 2024-03-29T11:39:12.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.645+0000 7f2ef6ffd640 10 client.4354.objecter ms_dispatch 0x7f2f08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:12.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.645+0000 7f2f0659c640 10 monclient: get_auth_request con 0x7f2ee806e7c0 auth_method 0 2024-03-29T11:39:12.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.645+0000 7f2ef6ffd640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:12.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.645+0000 7f2ef6ffd640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:12.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.645+0000 7f2ef6ffd640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-29T11:39:12.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.647+0000 7f2ef6ffd640 10 client.4354.objecter ms_handle_connect 0x7f2ee806e7c0 2024-03-29T11:39:12.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.654+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f00050a80 con 0x7f2f0813ee90 2024-03-29T11:39:12.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.654+0000 7f2ef6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:12.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.654+0000 7f2ef6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:12.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.986+0000 7f2f0cc63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:12.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.986+0000 7f2f0cc63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:12.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.986+0000 7f2f0cc63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:39:12.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.986+0000 7f2f0cc63640 1 -- 172.21.15.37:0/3909765109 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2eec005330 con 0x7f2f0813ee90 2024-03-29T11:39:12.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.988+0000 7f2ef6ffd640 1 -- 172.21.15.37:0/3909765109 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f2f00050c20 con 0x7f2f0813ee90 2024-03-29T11:39:12.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.988+0000 7f2ef6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:12.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.988+0000 7f2ef6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:12.990 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:39:12.991 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:12.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.992+0000 7f2f0cc63640 10 librados: watch_flush enter 2024-03-29T11:39:12.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.992+0000 7f2f0cc63640 10 librados: watch_flush exit 2024-03-29T11:39:12.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.993+0000 7f2f0cc63640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-29T11:39:12.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.993+0000 7f2f0cc63640 10 client.4354.objecter successfully canceled tick 2024-03-29T11:39:12.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.995+0000 7f2f0cc63640 1 -- 172.21.15.37:0/3909765109 >> v1:172.21.15.146:6816/2899149139 conn(0x7f2ee806e7c0 legacy=0x7f2ee8070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:12.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.995+0000 7f2f0cc63640 10 monclient: shutdown 2024-03-29T11:39:12.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.995+0000 7f2f0cc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:12.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.995+0000 7f2f0cc63640 1 -- 172.21.15.37:0/3909765109 >> v1:172.21.15.146:6789/0 conn(0x7f2f0813ee90 legacy=0x7f2f081ee7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:12.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.995+0000 7f2f0cc63640 1 -- 172.21.15.37:0/3909765109 shutdown_connections 2024-03-29T11:39:12.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.996+0000 7f2f0cc63640 1 -- 172.21.15.37:0/3909765109 >> 172.21.15.37:0/3909765109 conn(0x7f2f080b35e0 msgr2=0x7f2f081ccf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:12.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.996+0000 7f2f0cc63640 1 -- 172.21.15.37:0/3909765109 shutdown_connections 2024-03-29T11:39:12.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.996+0000 7f2f0cc63640 1 -- 172.21.15.37:0/3909765109 wait complete. 2024-03-29T11:39:12.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:12.996+0000 7f2f0cc63640 1 librados: shutdown 2024-03-29T11:39:13.014 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:19.821 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:19.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.966+0000 7f2de4617640 10 monclient: get_monmap_and_config 2024-03-29T11:39:19.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.966+0000 7f2de4617640 10 monclient: build_initial_monmap 2024-03-29T11:39:19.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.966+0000 7f2de4617640 10 monclient: monmap: 2024-03-29T11:39:19.968 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:19.968 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:19.968 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:39:19.967384+0000 2024-03-29T11:39:19.968 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:39:19.967384+0000 2024-03-29T11:39:19.968 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:19.968 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:19.968 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:19.968 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:39:19.968 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:39:19.968 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:19.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.967+0000 7f2de4617640 1 Processor -- start 2024-03-29T11:39:19.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.967+0000 7f2de4617640 1 -- start start 2024-03-29T11:39:19.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.967+0000 7f2de4617640 10 monclient: init 2024-03-29T11:39:19.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.968+0000 7f2de4617640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:19.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.968+0000 7f2de4617640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:19.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.968+0000 7f2de4617640 10 monclient(hunting): picked mon.noname-b con 0x7f2ddc1422b0 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:19.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.968+0000 7f2de4617640 10 monclient(hunting): picked mon.noname-a con 0x7f2ddc142690 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:19.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.968+0000 7f2de4617640 10 monclient(hunting): picked mon.noname-c con 0x7f2ddc0b17d0 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:19.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.968+0000 7f2de4617640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc142c80 con 0x7f2ddc142690 2024-03-29T11:39:19.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.968+0000 7f2de4617640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc0aae40 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.968+0000 7f2de4617640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc13fbd0 con 0x7f2ddc1422b0 2024-03-29T11:39:19.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.968+0000 7f2de4617640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:19.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.968+0000 7f2de4617640 10 monclient(hunting): authenticate will time out at 1139.417649s 2024-03-29T11:39:19.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.970+0000 7f2de23b3640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f2ddc1422b0 0x7f2ddc13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:53270/0 (socket says 172.21.15.37:53270) 2024-03-29T11:39:19.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.970+0000 7f2de23b3640 1 -- 172.21.15.37:0/3083770358 learned_addr learned my addr 172.21.15.37:0/3083770358 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:19.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.970+0000 7f2de13b1640 1 -- 172.21.15.37:0/3083770358 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323578842 0 0) 0x7f2ddc13fbd0 con 0x7f2ddc1422b0 2024-03-29T11:39:19.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.970+0000 7f2de13b1640 10 monclient(hunting): _negotiate 2024-03-29T11:39:19.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.970+0000 7f2de13b1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:19.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.970+0000 7f2de13b1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:19.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.970+0000 7f2de13b1640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:39:19.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.970+0000 7f2de13b1640 1 -- 172.21.15.37:0/3083770358 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc800bd30 con 0x7f2ddc1422b0 2024-03-29T11:39:19.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.971+0000 7f2de13b1640 1 -- 172.21.15.37:0/3083770358 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3302704341 0 0) 0x7f2dc800bd30 con 0x7f2ddc1422b0 2024-03-29T11:39:19.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.971+0000 7f2de13b1640 1 -- 172.21.15.37:0/3083770358 >> v1:172.21.15.146:6789/0 conn(0x7f2ddc0b17d0 legacy=0x7f2ddc0b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:19.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.973+0000 7f2de13b1640 1 -- 172.21.15.37:0/3083770358 >> v1:172.21.15.37:6789/0 conn(0x7f2ddc142690 legacy=0x7f2ddc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:19.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.973+0000 7f2de13b1640 10 monclient: _finish_hunting 0 2024-03-29T11:39:19.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.973+0000 7f2de13b1640 1 monclient: found mon.noname-b 2024-03-29T11:39:19.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.973+0000 7f2de13b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:19.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.973+0000 7f2de13b1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.973+0000 7f2de13b1640 1 -- 172.21.15.37:0/3083770358 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc151940 con 0x7f2ddc1422b0 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.973+0000 7f2de13b1640 10 monclient: _finish_auth 0 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.973+0000 7f2de13b1640 10 monclient: _check_auth_tickets 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.973+0000 7f2de13b1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.973+0000 7f2de13b1640 1 -- 172.21.15.37:0/3083770358 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2dd4001a10 con 0x7f2ddc1422b0 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.973+0000 7f2de13b1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.973+0000 7f2de13b1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.973+0000 7f2de13b1640 10 monclient: dump: 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:19.976 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:19.977 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:19.977 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:19.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.975+0000 7f2de4617640 5 monclient: authenticate success, global_id 4400 2024-03-29T11:39:19.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.975+0000 7f2de4617640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:19.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.976+0000 7f2de13b1640 1 -- 172.21.15.37:0/3083770358 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dd4001b70 con 0x7f2ddc1422b0 2024-03-29T11:39:19.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.976+0000 7f2de13b1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:19.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.976+0000 7f2de4617640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:19.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.976+0000 7f2de4617640 4 set_mon_vals no callback set 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.976+0000 7f2de13b1640 1 -- 172.21.15.37:0/3083770358 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2dd4001fc0 con 0x7f2ddc1422b0 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.976+0000 7f2de13b1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.976+0000 7f2de13b1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.976+0000 7f2de13b1640 10 monclient: dump: 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.977+0000 7f2de4617640 10 monclient: shutdown 2024-03-29T11:39:19.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.977+0000 7f2de4617640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:19.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.977+0000 7f2de4617640 1 -- 172.21.15.37:0/3083770358 >> v1:172.21.15.37:6790/0 conn(0x7f2ddc1422b0 legacy=0x7f2ddc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:19.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.977+0000 7f2de4617640 1 -- 172.21.15.37:0/3083770358 shutdown_connections 2024-03-29T11:39:19.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.977+0000 7f2de4617640 1 -- 172.21.15.37:0/3083770358 >> 172.21.15.37:0/3083770358 conn(0x7f2ddc066a00 msgr2=0x7f2ddc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:19.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.977+0000 7f2de4617640 1 -- 172.21.15.37:0/3083770358 shutdown_connections 2024-03-29T11:39:19.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.977+0000 7f2de4617640 1 -- 172.21.15.37:0/3083770358 wait complete. 2024-03-29T11:39:19.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.981+0000 7f2de4617640 10 monclient: build_initial_monmap 2024-03-29T11:39:19.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.981+0000 7f2de4617640 10 monclient: monmap: 2024-03-29T11:39:19.982 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:19.982 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:19.982 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:39:19.982 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:39:19.982 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:19.983 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:19.983 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:19.983 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:39:19.983 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:39:19.983 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:19.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.982+0000 7f2de4617640 1 librados: starting msgr at 2024-03-29T11:39:19.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.982+0000 7f2de4617640 1 librados: starting objecter 2024-03-29T11:39:19.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.982+0000 7f2de4617640 1 Processor -- start 2024-03-29T11:39:19.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.982+0000 7f2de4617640 1 -- start start 2024-03-29T11:39:19.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.982+0000 7f2de4617640 1 librados: setting wanted keys 2024-03-29T11:39:19.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.982+0000 7f2de4617640 1 librados: calling monclient init 2024-03-29T11:39:19.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.982+0000 7f2de4617640 10 monclient: init 2024-03-29T11:39:19.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.983+0000 7f2de4617640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:19.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.983+0000 7f2de4617640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:19.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.983+0000 7f2de4617640 10 monclient(hunting): picked mon.noname-a con 0x7f2ddc0b17d0 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:19.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.983+0000 7f2de4617640 10 monclient(hunting): picked mon.noname-c con 0x7f2ddc1422b0 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:19.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.983+0000 7f2de4617640 10 monclient(hunting): picked mon.noname-b con 0x7f2ddc142690 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:19.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.983+0000 7f2de4617640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc0ad090 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.983+0000 7f2de4617640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc071e60 con 0x7f2ddc142690 2024-03-29T11:39:19.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.983+0000 7f2de4617640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc071ff0 con 0x7f2ddc1422b0 2024-03-29T11:39:19.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.983+0000 7f2de4617640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:19.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.983+0000 7f2de4617640 10 monclient(hunting): authenticate will time out at 1139.432042s 2024-03-29T11:39:19.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.986+0000 7f2de1bb2640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f2ddc1422b0 0x7f2ddc0b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:54072/0 (socket says 172.21.15.37:54072) 2024-03-29T11:39:19.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.986+0000 7f2de1bb2640 1 -- 172.21.15.37:0/1905334221 learned_addr learned my addr 172.21.15.37:0/1905334221 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:19.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.986+0000 7f2dceffd640 10 client.?.objecter ms_handle_connect 0x7f2ddc1422b0 2024-03-29T11:39:19.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.986+0000 7f2dceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:19.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 10 client.?.objecter ms_handle_connect 0x7f2ddc0b17d0 2024-03-29T11:39:19.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:19.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929384906 0 0) 0x7f2ddc0ad090 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:19.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:19.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:19.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:39:19.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc0004eb0 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3417175317 0 0) 0x7f2dc0004eb0 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ddc0ad090 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2dd4001e80 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 10 monclient(hunting): dump: 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.987+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164887736 0 0) 0x7f2ddc071ff0 con 0x7f2ddc1422b0 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.988+0000 7f2dceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.988+0000 7f2dceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.988+0000 7f2dceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.988+0000 7f2dceffd640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.988+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc0004eb0 con 0x7f2ddc1422b0 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.988+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 857826673 0 0) 0x7f2ddc0ad090 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.988+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 >> v1:172.21.15.37:6790/0 conn(0x7f2ddc1422b0 legacy=0x7f2ddc0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:19.991 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.988+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 >> v1:172.21.15.146:6789/0 conn(0x7f2ddc142690 legacy=0x7f2ddc0b0cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:19.992 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.989+0000 7f2dceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.989+0000 7f2dceffd640 1 monclient: found mon.a 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.989+0000 7f2dceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.989+0000 7f2dceffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.989+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc071780 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.990+0000 7f2dceffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.990+0000 7f2dceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.990+0000 7f2dceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.990+0000 7f2de4617640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.990+0000 7f2de4617640 10 monclient: _renew_subs 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.990+0000 7f2de4617640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.990+0000 7f2de4617640 1 -- 172.21.15.37:0/1905334221 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ddc0719d0 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.990+0000 7f2de4617640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.990+0000 7f2de4617640 10 monclient: _renew_subs 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.990+0000 7f2de4617640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:39:19.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.990+0000 7f2de4617640 1 -- 172.21.15.37:0/1905334221 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ddc0ab8e0 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.990+0000 7f2de4617640 1 librados: init done 2024-03-29T11:39:19.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.991+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dd4004f70 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.991+0000 7f2dceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:19.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.991+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2dd4005280 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.991+0000 7f2dceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:19.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.991+0000 7f2dceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:19.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.991+0000 7f2dceffd640 10 monclient: dump: 2024-03-29T11:39:19.994 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:19.994 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:19.994 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:19.994 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:19.994 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:19.994 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:19.995 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:19.995 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:19.995 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:19.995 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:19.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.991+0000 7f2de0bb0640 4 set_mon_vals no callback set 2024-03-29T11:39:19.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.992+0000 7f2de4617640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:19.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.992+0000 7f2de4617640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:19.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.992+0000 7f2de4617640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:39:19.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.992+0000 7f2de4617640 1 -- 172.21.15.37:0/1905334221 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc4005080 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.994+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f2dd401b250 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.994+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f2dd4084f70 con 0x7f2ddc0b17d0 2024-03-29T11:39:19.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.994+0000 7f2dceffd640 10 client.4377.objecter ms_dispatch 0x7f2ddc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:19.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.994+0000 7f2dceffd640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:19.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.994+0000 7f2dceffd640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:19.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.995+0000 7f2dceffd640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-29T11:39:19.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.998+0000 7f2de1bb2640 10 monclient: get_auth_request con 0x7f2dc006dd30 auth_method 0 2024-03-29T11:39:20.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:19.999+0000 7f2dceffd640 10 client.4377.objecter ms_handle_connect 0x7f2dc006dd30 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.000+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2dd4050850 con 0x7f2ddc0b17d0 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.000+0000 7f2dceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.000+0000 7f2dceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:20.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.338+0000 7f2de4617640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:20.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.338+0000 7f2de4617640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:20.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.338+0000 7f2de4617640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:39:20.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.338+0000 7f2de4617640 1 -- 172.21.15.37:0/1905334221 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dc4005330 con 0x7f2ddc0b17d0 2024-03-29T11:39:20.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.340+0000 7f2dceffd640 1 -- 172.21.15.37:0/1905334221 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f2dd4052780 con 0x7f2ddc0b17d0 2024-03-29T11:39:20.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.340+0000 7f2dceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:20.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.340+0000 7f2dceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:20.341 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:39:20.341 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:20.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.343+0000 7f2de4617640 10 librados: watch_flush enter 2024-03-29T11:39:20.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.343+0000 7f2de4617640 10 librados: watch_flush exit 2024-03-29T11:39:20.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.343+0000 7f2de4617640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-29T11:39:20.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.343+0000 7f2de4617640 10 client.4377.objecter successfully canceled tick 2024-03-29T11:39:20.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.344+0000 7f2de4617640 1 -- 172.21.15.37:0/1905334221 >> v1:172.21.15.146:6816/2899149139 conn(0x7f2dc006dd30 legacy=0x7f2dc00701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:20.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.344+0000 7f2de4617640 10 monclient: shutdown 2024-03-29T11:39:20.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.344+0000 7f2de4617640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:20.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.344+0000 7f2de4617640 1 -- 172.21.15.37:0/1905334221 >> v1:172.21.15.37:6789/0 conn(0x7f2ddc0b17d0 legacy=0x7f2ddc139780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:20.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.344+0000 7f2de4617640 1 -- 172.21.15.37:0/1905334221 shutdown_connections 2024-03-29T11:39:20.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.344+0000 7f2de4617640 1 -- 172.21.15.37:0/1905334221 >> 172.21.15.37:0/1905334221 conn(0x7f2ddc066a00 msgr2=0x7f2ddc0b3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:20.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.344+0000 7f2de4617640 1 -- 172.21.15.37:0/1905334221 shutdown_connections 2024-03-29T11:39:20.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.344+0000 7f2de4617640 1 -- 172.21.15.37:0/1905334221 wait complete. 2024-03-29T11:39:20.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:20.344+0000 7f2de4617640 1 librados: shutdown 2024-03-29T11:39:20.361 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:27.168 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:27.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.323+0000 7f8da3d8f640 10 monclient: get_monmap_and_config 2024-03-29T11:39:27.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.323+0000 7f8da3d8f640 10 monclient: build_initial_monmap 2024-03-29T11:39:27.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.323+0000 7f8da3d8f640 10 monclient: monmap: 2024-03-29T11:39:27.325 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:27.325 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:27.325 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:39:27.324381+0000 2024-03-29T11:39:27.325 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:39:27.324381+0000 2024-03-29T11:39:27.325 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:27.325 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:27.325 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:27.325 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:39:27.325 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:39:27.325 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:27.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.324+0000 7f8da3d8f640 1 Processor -- start 2024-03-29T11:39:27.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.325+0000 7f8da3d8f640 1 -- start start 2024-03-29T11:39:27.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.325+0000 7f8da3d8f640 10 monclient: init 2024-03-29T11:39:27.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.327+0000 7f8da3d8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:27.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.327+0000 7f8da3d8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:27.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.327+0000 7f8da3d8f640 10 monclient(hunting): picked mon.noname-a con 0x7f8d9c0b0770 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:27.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.327+0000 7f8da3d8f640 10 monclient(hunting): picked mon.noname-b con 0x7f8d9c14a700 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:27.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.327+0000 7f8da3d8f640 10 monclient(hunting): picked mon.noname-c con 0x7f8d9c0ac5a0 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:27.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.327+0000 7f8da3d8f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9c0aba70 con 0x7f8d9c0b0770 2024-03-29T11:39:27.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.327+0000 7f8da3d8f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9c0aecb0 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.327+0000 7f8da3d8f640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9c0af150 con 0x7f8d9c14a700 2024-03-29T11:39:27.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.327+0000 7f8da3d8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:27.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.327+0000 7f8da3d8f640 10 monclient(hunting): authenticate will time out at 1146.776822s 2024-03-29T11:39:27.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.328+0000 7f8da1b2b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8d9c0b0770 0x7f8d9c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:56908/0 (socket says 172.21.15.37:56908) 2024-03-29T11:39:27.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.328+0000 7f8da1b2b640 1 -- 172.21.15.37:0/1316568393 learned_addr learned my addr 172.21.15.37:0/1316568393 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:27.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.329+0000 7f8da0b29640 1 -- 172.21.15.37:0/1316568393 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538701178 0 0) 0x7f8d9c0aba70 con 0x7f8d9c0b0770 2024-03-29T11:39:27.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.329+0000 7f8da0b29640 10 monclient(hunting): _negotiate 2024-03-29T11:39:27.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.329+0000 7f8da0b29640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:27.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.329+0000 7f8da0b29640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:27.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.329+0000 7f8da0b29640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:39:27.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.329+0000 7f8da0b29640 1 -- 172.21.15.37:0/1316568393 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d88003520 con 0x7f8d9c0b0770 2024-03-29T11:39:27.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 1 -- 172.21.15.37:0/1316568393 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4289574413 0 0) 0x7f8d88003520 con 0x7f8d9c0b0770 2024-03-29T11:39:27.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 1 -- 172.21.15.37:0/1316568393 >> v1:172.21.15.37:6790/0 conn(0x7f8d9c14a700 legacy=0x7f8d9c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 1 -- 172.21.15.37:0/1316568393 >> v1:172.21.15.146:6789/0 conn(0x7f8d9c0ac5a0 legacy=0x7f8d9c151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:27.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 10 monclient: _finish_hunting 0 2024-03-29T11:39:27.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 1 monclient: found mon.noname-a 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 1 -- 172.21.15.37:0/1316568393 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9c0ac9d0 con 0x7f8d9c0b0770 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 10 monclient: _finish_auth 0 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 10 monclient: _check_auth_tickets 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 1 -- 172.21.15.37:0/1316568393 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8d98001a10 con 0x7f8d9c0b0770 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da0b29640 10 monclient: dump: 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:27.333 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:27.334 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:27.334 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:27.334 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:27.334 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:27.334 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:27.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da3d8f640 5 monclient: authenticate success, global_id 4380 2024-03-29T11:39:27.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.331+0000 7f8da3d8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:27.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.332+0000 7f8da0b29640 1 -- 172.21.15.37:0/1316568393 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d98001ca0 con 0x7f8d9c0b0770 2024-03-29T11:39:27.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.332+0000 7f8da0b29640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:27.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.332+0000 7f8da3d8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:27.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.332+0000 7f8da3d8f640 4 set_mon_vals no callback set 2024-03-29T11:39:27.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.332+0000 7f8da0b29640 1 -- 172.21.15.37:0/1316568393 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8d98002030 con 0x7f8d9c0b0770 2024-03-29T11:39:27.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.332+0000 7f8da0b29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:27.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.332+0000 7f8da0b29640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:27.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.332+0000 7f8da0b29640 10 monclient: dump: 2024-03-29T11:39:27.335 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:27.335 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:27.335 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:27.335 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:27.336 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:27.336 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:27.336 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:27.336 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:27.336 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:27.336 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:27.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.333+0000 7f8da3d8f640 10 monclient: shutdown 2024-03-29T11:39:27.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.333+0000 7f8da3d8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:27.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.333+0000 7f8da3d8f640 1 -- 172.21.15.37:0/1316568393 >> v1:172.21.15.37:6789/0 conn(0x7f8d9c0b0770 legacy=0x7f8d9c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.333+0000 7f8da3d8f640 1 -- 172.21.15.37:0/1316568393 shutdown_connections 2024-03-29T11:39:27.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.333+0000 7f8da3d8f640 1 -- 172.21.15.37:0/1316568393 >> 172.21.15.37:0/1316568393 conn(0x7f8d9c06b660 msgr2=0x7f8d9c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:27.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.333+0000 7f8da3d8f640 1 -- 172.21.15.37:0/1316568393 shutdown_connections 2024-03-29T11:39:27.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.333+0000 7f8da3d8f640 1 -- 172.21.15.37:0/1316568393 wait complete. 2024-03-29T11:39:27.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.334+0000 7f8da3d8f640 10 monclient: build_initial_monmap 2024-03-29T11:39:27.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.334+0000 7f8da3d8f640 10 monclient: monmap: 2024-03-29T11:39:27.341 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:27.341 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:27.341 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:39:27.341 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:39:27.341 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:27.341 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:27.341 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:27.341 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:39:27.341 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:39:27.341 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:27.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.335+0000 7f8da3d8f640 1 librados: starting msgr at 2024-03-29T11:39:27.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.335+0000 7f8da3d8f640 1 librados: starting objecter 2024-03-29T11:39:27.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.335+0000 7f8da3d8f640 1 Processor -- start 2024-03-29T11:39:27.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.335+0000 7f8da3d8f640 1 -- start start 2024-03-29T11:39:27.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.335+0000 7f8da3d8f640 1 librados: setting wanted keys 2024-03-29T11:39:27.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.335+0000 7f8da3d8f640 1 librados: calling monclient init 2024-03-29T11:39:27.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.335+0000 7f8da3d8f640 10 monclient: init 2024-03-29T11:39:27.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.336+0000 7f8da3d8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:27.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.336+0000 7f8da3d8f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:27.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.336+0000 7f8da3d8f640 10 monclient(hunting): picked mon.noname-c con 0x7f8d9c0ac5a0 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:27.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.336+0000 7f8da3d8f640 10 monclient(hunting): picked mon.noname-b con 0x7f8d9c0b0770 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:27.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.336+0000 7f8da3d8f640 10 monclient(hunting): picked mon.noname-a con 0x7f8d9c14a700 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:27.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.336+0000 7f8da3d8f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9c126140 con 0x7f8d9c14a700 2024-03-29T11:39:27.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.336+0000 7f8da3d8f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9c145ac0 con 0x7f8d9c0b0770 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.337+0000 7f8da1b2b640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f8d9c0ac5a0 0x7f8d9c144ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:38132/0 (socket says 172.21.15.37:38132) 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.337+0000 7f8da1b2b640 1 -- 172.21.15.37:0/442569799 learned_addr learned my addr 172.21.15.37:0/442569799 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.337+0000 7f8da3d8f640 1 -- 172.21.15.37:0/442569799 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9c145c50 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.337+0000 7f8da3d8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.337+0000 7f8da3d8f640 10 monclient(hunting): authenticate will time out at 1146.786286s 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.337+0000 7f8d927fc640 10 client.?.objecter ms_handle_connect 0x7f8d9c0ac5a0 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.337+0000 7f8d927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.337+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699933272 0 0) 0x7f8d9c145c50 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.337+0000 7f8d927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.337+0000 7f8d927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.337+0000 7f8d927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.337+0000 7f8d927fc640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.338+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d80004eb0 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.338+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3731742547 0 0) 0x7f8d80004eb0 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.338+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d9c145c50 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 10 client.?.objecter ms_handle_connect 0x7f8d9c14a700 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8d980018d0 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 10 monclient(hunting): dump: 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929296617 0 0) 0x7f8d9c126140 con 0x7f8d9c14a700 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d80004eb0 con 0x7f8d9c14a700 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 190802411 0 0) 0x7f8d9c145c50 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.339+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 >> v1:172.21.15.146:6789/0 conn(0x7f8d9c0b0770 legacy=0x7f8d9c126660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:27.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 >> v1:172.21.15.37:6789/0 conn(0x7f8d9c14a700 legacy=0x7f8d9c1e6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:27.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 1 monclient: found mon.c 2024-03-29T11:39:27.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:27.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:27.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9c125cb0 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8da3d8f640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8da3d8f640 10 monclient: _renew_subs 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8da3d8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8da3d8f640 1 -- 172.21.15.37:0/442569799 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9c125ea0 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8da3d8f640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8da3d8f640 10 monclient: _renew_subs 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8da3d8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8da3d8f640 1 -- 172.21.15.37:0/442569799 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9c1df7a0 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8da3d8f640 1 librados: init done 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d98001a30 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8d98005190 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d93fff640 4 set_mon_vals no callback set 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.342+0000 7f8d927fc640 10 monclient: dump: 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.344+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f8d980052f0 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.345+0000 7f8da3d8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:27.350 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.345+0000 7f8da3d8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.345+0000 7f8da3d8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.345+0000 7f8da3d8f640 1 -- 172.21.15.37:0/442569799 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d84005080 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.346+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f8d98084fe0 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.346+0000 7f8da132a640 10 monclient: get_auth_request con 0x7f8d8006dac0 auth_method 0 2024-03-29T11:39:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.346+0000 7f8d927fc640 10 client.4409.objecter ms_dispatch 0x7f8d9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.346+0000 7f8d927fc640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.346+0000 7f8d927fc640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.346+0000 7f8d927fc640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T11:39:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.346+0000 7f8d927fc640 10 client.4409.objecter ms_handle_connect 0x7f8d8006dac0 2024-03-29T11:39:27.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.358+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d980508c0 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.358+0000 7f8d927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:27.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.358+0000 7f8d927fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:27.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.705+0000 7f8da3d8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:27.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.705+0000 7f8da3d8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:27.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.705+0000 7f8da3d8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:27.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.705+0000 7f8da3d8f640 1 -- 172.21.15.37:0/442569799 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d84005330 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.706+0000 7f8d927fc640 1 -- 172.21.15.37:0/442569799 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f8d980527f0 con 0x7f8d9c0ac5a0 2024-03-29T11:39:27.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.706+0000 7f8d927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:27.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.706+0000 7f8d927fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:27.708 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:39:27.708 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:27.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.709+0000 7f8da3d8f640 10 librados: watch_flush enter 2024-03-29T11:39:27.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.709+0000 7f8da3d8f640 10 librados: watch_flush exit 2024-03-29T11:39:27.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.709+0000 7f8da3d8f640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T11:39:27.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.709+0000 7f8da3d8f640 10 client.4409.objecter successfully canceled tick 2024-03-29T11:39:27.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.709+0000 7f8da3d8f640 1 -- 172.21.15.37:0/442569799 >> v1:172.21.15.146:6816/2899149139 conn(0x7f8d8006dac0 legacy=0x7f8d8006ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.710+0000 7f8da3d8f640 10 monclient: shutdown 2024-03-29T11:39:27.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.710+0000 7f8da3d8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:27.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.710+0000 7f8da3d8f640 1 -- 172.21.15.37:0/442569799 >> v1:172.21.15.37:6790/0 conn(0x7f8d9c0ac5a0 legacy=0x7f8d9c144ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.710+0000 7f8da3d8f640 1 -- 172.21.15.37:0/442569799 shutdown_connections 2024-03-29T11:39:27.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.710+0000 7f8da3d8f640 1 -- 172.21.15.37:0/442569799 >> 172.21.15.37:0/442569799 conn(0x7f8d9c06b660 msgr2=0x7f8d9c0abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:27.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.710+0000 7f8da3d8f640 1 -- 172.21.15.37:0/442569799 shutdown_connections 2024-03-29T11:39:27.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.710+0000 7f8da3d8f640 1 -- 172.21.15.37:0/442569799 wait complete. 2024-03-29T11:39:27.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:27.710+0000 7f8da3d8f640 1 librados: shutdown 2024-03-29T11:39:27.727 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:34.534 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:34.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.680+0000 7fbbef457640 10 monclient: get_monmap_and_config 2024-03-29T11:39:34.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.680+0000 7fbbef457640 10 monclient: build_initial_monmap 2024-03-29T11:39:34.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.680+0000 7fbbef457640 10 monclient: monmap: 2024-03-29T11:39:34.682 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:34.682 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:34.682 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:39:34.681371+0000 2024-03-29T11:39:34.682 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:39:34.681371+0000 2024-03-29T11:39:34.682 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:34.682 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:34.683 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:34.683 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:39:34.683 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:39:34.683 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:34.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.681+0000 7fbbef457640 1 Processor -- start 2024-03-29T11:39:34.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.681+0000 7fbbef457640 1 -- start start 2024-03-29T11:39:34.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.681+0000 7fbbef457640 10 monclient: init 2024-03-29T11:39:34.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.682+0000 7fbbef457640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:34.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.682+0000 7fbbef457640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:34.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.682+0000 7fbbef457640 10 monclient(hunting): picked mon.noname-b con 0x7fbbe81457f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.682+0000 7fbbef457640 10 monclient(hunting): picked mon.noname-a con 0x7fbbe813f650 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.682+0000 7fbbef457640 10 monclient(hunting): picked mon.noname-c con 0x7fbbe80a9d70 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.682+0000 7fbbef457640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe80aaef0 con 0x7fbbe813f650 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.682+0000 7fbbef457640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe8145bd0 con 0x7fbbe80a9d70 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.682+0000 7fbbef457640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe8145eb0 con 0x7fbbe81457f0 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.682+0000 7fbbef457640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.682+0000 7fbbef457640 10 monclient(hunting): authenticate will time out at 1154.131915s 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.683+0000 7fbbed9f4640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbbe80a9d70 0x7fbbe814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:45016/0 (socket says 172.21.15.37:45016) 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.683+0000 7fbbed9f4640 1 -- 172.21.15.37:0/3321769753 learned_addr learned my addr 172.21.15.37:0/3321769753 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 1 -- 172.21.15.37:0/3321769753 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660349081 0 0) 0x7fbbe8145bd0 con 0x7fbbe80a9d70 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 1 -- 172.21.15.37:0/3321769753 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd4003520 con 0x7fbbe80a9d70 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 1 -- 172.21.15.37:0/3321769753 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076801798 0 0) 0x7fbbe80aaef0 con 0x7fbbe813f650 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 1 -- 172.21.15.37:0/3321769753 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbe8145bd0 con 0x7fbbe813f650 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 1 -- 172.21.15.37:0/3321769753 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286765889 0 0) 0x7fbbe8145eb0 con 0x7fbbe81457f0 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 1 -- 172.21.15.37:0/3321769753 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbe80aaef0 con 0x7fbbe81457f0 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 1 -- 172.21.15.37:0/3321769753 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 157576681 0 0) 0x7fbbd4003520 con 0x7fbbe80a9d70 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 1 -- 172.21.15.37:0/3321769753 >> v1:172.21.15.37:6790/0 conn(0x7fbbe81457f0 legacy=0x7fbbe813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 1 -- 172.21.15.37:0/3321769753 >> v1:172.21.15.37:6789/0 conn(0x7fbbe813f650 legacy=0x7fbbe80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 1 monclient: found mon.noname-c 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.684+0000 7fbbdffff640 1 -- 172.21.15.37:0/3321769753 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe8146340 con 0x7fbbe80a9d70 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbdffff640 10 monclient: _finish_auth 0 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbdffff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbdffff640 1 -- 172.21.15.37:0/3321769753 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbbe0001a10 con 0x7fbbe80a9d70 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbdffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbdffff640 10 monclient: dump: 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbef457640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbef457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbdffff640 1 -- 172.21.15.37:0/3321769753 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbe0001c30 con 0x7fbbe80a9d70 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbdffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbdffff640 1 -- 172.21.15.37:0/3321769753 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbbe0001fc0 con 0x7fbbe80a9d70 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbdffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbdffff640 10 monclient: dump: 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbef457640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.685+0000 7fbbef457640 4 set_mon_vals no callback set 2024-03-29T11:39:34.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.686+0000 7fbbef457640 10 monclient: shutdown 2024-03-29T11:39:34.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.686+0000 7fbbef457640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.686+0000 7fbbef457640 1 -- 172.21.15.37:0/3321769753 >> v1:172.21.15.146:6789/0 conn(0x7fbbe80a9d70 legacy=0x7fbbe814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.687+0000 7fbbef457640 1 -- 172.21.15.37:0/3321769753 shutdown_connections 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.687+0000 7fbbef457640 1 -- 172.21.15.37:0/3321769753 >> 172.21.15.37:0/3321769753 conn(0x7fbbe8066a00 msgr2=0x7fbbe8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.688+0000 7fbbef457640 1 -- 172.21.15.37:0/3321769753 shutdown_connections 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.688+0000 7fbbef457640 1 -- 172.21.15.37:0/3321769753 wait complete. 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.689+0000 7fbbef457640 10 monclient: build_initial_monmap 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.689+0000 7fbbef457640 10 monclient: monmap: 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.689+0000 7fbbef457640 1 librados: starting msgr at 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.689+0000 7fbbef457640 1 librados: starting objecter 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.689+0000 7fbbef457640 1 Processor -- start 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.690+0000 7fbbef457640 1 -- start start 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.690+0000 7fbbef457640 1 librados: setting wanted keys 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.690+0000 7fbbef457640 1 librados: calling monclient init 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.690+0000 7fbbef457640 10 monclient: init 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.690+0000 7fbbef457640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.690+0000 7fbbef457640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.690+0000 7fbbef457640 10 monclient(hunting): picked mon.noname-c con 0x7fbbe80a9d70 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbed1f3640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fbbe80a9d70 0x7fbbe8142740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:55452/0 (socket says 172.21.15.37:55452) 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbed1f3640 1 -- 172.21.15.37:0/3774010357 learned_addr learned my addr 172.21.15.37:0/3774010357 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbef457640 10 monclient(hunting): picked mon.noname-a con 0x7fbbe813f650 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbddffb640 10 client.?.objecter ms_handle_connect 0x7fbbe80a9d70 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbef457640 10 monclient(hunting): picked mon.noname-b con 0x7fbbe81457f0 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbef457640 1 -- 172.21.15.37:0/3774010357 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe81402c0 con 0x7fbbe813f650 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbef457640 1 -- 172.21.15.37:0/3774010357 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe813fe30 con 0x7fbbe81457f0 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbef457640 1 -- 172.21.15.37:0/3774010357 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe8142e50 con 0x7fbbe80a9d70 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbef457640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbef457640 10 monclient(hunting): authenticate will time out at 1154.140320s 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841358476 0 0) 0x7fbbe8142e50 con 0x7fbbe80a9d70 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbddffb640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.691+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbcc004eb0 con 0x7fbbe80a9d70 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.692+0000 7fbbddffb640 10 client.?.objecter ms_handle_connect 0x7fbbe813f650 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.692+0000 7fbbddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.692+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 442142926 0 0) 0x7fbbcc004eb0 con 0x7fbbe80a9d70 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.692+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbe8142e50 con 0x7fbbe80a9d70 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.692+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbbe4001950 con 0x7fbbe80a9d70 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.692+0000 7fbbddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.692+0000 7fbbddffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.692+0000 7fbbddffb640 10 monclient(hunting): dump: 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.692+0000 7fbbddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142751871 0 0) 0x7fbbd8001130 con 0x7fbbe813f650 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbcc004eb0 con 0x7fbbe813f650 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 10 client.?.objecter ms_handle_connect 0x7fbbe81457f0 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383501514 0 0) 0x7fbbe813fe30 con 0x7fbbe81457f0 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd8001130 con 0x7fbbe81457f0 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 412906928 0 0) 0x7fbbcc004eb0 con 0x7fbbe813f650 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbe813fe30 con 0x7fbbe813f650 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbbd8001830 con 0x7fbbe813f650 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.693+0000 7fbbddffb640 10 monclient(hunting): dump: 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3154171151 0 0) 0x7fbbe8142e50 con 0x7fbbe80a9d70 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 >> v1:172.21.15.146:6789/0 conn(0x7fbbe81457f0 legacy=0x7fbbe81e6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 >> v1:172.21.15.37:6789/0 conn(0x7fbbe813f650 legacy=0x7fbbe8140480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbddffb640 1 monclient: found mon.c 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe80b16d0 con 0x7fbbe80a9d70 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbddffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbef457640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbef457640 10 monclient: _renew_subs 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbef457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbef457640 1 -- 172.21.15.37:0/3774010357 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe813b570 con 0x7fbbe80a9d70 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbef457640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbef457640 10 monclient: _renew_subs 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbef457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbef457640 1 -- 172.21.15.37:0/3774010357 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe8140d70 con 0x7fbbe80a9d70 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.694+0000 7fbbef457640 1 librados: init done 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.695+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbe4001c50 con 0x7fbbe80a9d70 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.695+0000 7fbbddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.695+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbbe4003a40 con 0x7fbbe80a9d70 2024-03-29T11:39:34.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.695+0000 7fbbddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:34.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.695+0000 7fbbddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:34.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.695+0000 7fbbdeffd640 4 set_mon_vals no callback set 2024-03-29T11:39:34.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.695+0000 7fbbddffb640 10 monclient: dump: 2024-03-29T11:39:34.706 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:34.706 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:34.706 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:34.706 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:34.706 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:34.706 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:34.706 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:34.706 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:34.706 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:34.706 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:34.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.696+0000 7fbbef457640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:34.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.696+0000 7fbbef457640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:34.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.696+0000 7fbbef457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:34.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.696+0000 7fbbef457640 1 -- 172.21.15.37:0/3774010357 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc8005080 con 0x7fbbe80a9d70 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.698+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fbbe4003ba0 con 0x7fbbe80a9d70 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.699+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fbbe4084aa0 con 0x7fbbe80a9d70 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.699+0000 7fbbddffb640 10 client.4415.objecter ms_dispatch 0x7fbbe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.699+0000 7fbbddffb640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.699+0000 7fbbddffb640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.699+0000 7fbbddffb640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.699+0000 7fbbec9f2640 10 monclient: get_auth_request con 0x7fbbcc06df70 auth_method 0 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.699+0000 7fbbddffb640 10 client.4415.objecter ms_handle_connect 0x7fbbcc06df70 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.706+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbbe40503e0 con 0x7fbbe80a9d70 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.706+0000 7fbbddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:34.706+0000 7fbbddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:35.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.080+0000 7fbbef457640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:35.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.081+0000 7fbbef457640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:35.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.081+0000 7fbbef457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:35.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.081+0000 7fbbef457640 1 -- 172.21.15.37:0/3774010357 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbc8005330 con 0x7fbbe80a9d70 2024-03-29T11:39:35.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.082+0000 7fbbddffb640 1 -- 172.21.15.37:0/3774010357 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fbbe4052310 con 0x7fbbe80a9d70 2024-03-29T11:39:35.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.082+0000 7fbbddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:35.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.082+0000 7fbbddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:35.084 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:39:35.084 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:35.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.086+0000 7fbbef457640 10 librados: watch_flush enter 2024-03-29T11:39:35.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.086+0000 7fbbef457640 10 librados: watch_flush exit 2024-03-29T11:39:35.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.086+0000 7fbbef457640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-29T11:39:35.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.086+0000 7fbbef457640 10 client.4415.objecter successfully canceled tick 2024-03-29T11:39:35.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.086+0000 7fbbef457640 1 -- 172.21.15.37:0/3774010357 >> v1:172.21.15.146:6816/2899149139 conn(0x7fbbcc06df70 legacy=0x7fbbcc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:35.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.086+0000 7fbbef457640 10 monclient: shutdown 2024-03-29T11:39:35.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.086+0000 7fbbef457640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:35.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.086+0000 7fbbef457640 1 -- 172.21.15.37:0/3774010357 >> v1:172.21.15.37:6790/0 conn(0x7fbbe80a9d70 legacy=0x7fbbe8142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:35.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.086+0000 7fbbef457640 1 -- 172.21.15.37:0/3774010357 shutdown_connections 2024-03-29T11:39:35.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.086+0000 7fbbef457640 1 -- 172.21.15.37:0/3774010357 >> 172.21.15.37:0/3774010357 conn(0x7fbbe8066a00 msgr2=0x7fbbe80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:35.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.087+0000 7fbbef457640 1 -- 172.21.15.37:0/3774010357 shutdown_connections 2024-03-29T11:39:35.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.087+0000 7fbbef457640 1 -- 172.21.15.37:0/3774010357 wait complete. 2024-03-29T11:39:35.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:35.087+0000 7fbbef457640 1 librados: shutdown 2024-03-29T11:39:35.102 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:41.909 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:42.045 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.044+0000 7f4844693640 10 monclient: get_monmap_and_config 2024-03-29T11:39:42.045 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.044+0000 7f4844693640 10 monclient: build_initial_monmap 2024-03-29T11:39:42.045 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.044+0000 7f4844693640 10 monclient: monmap: 2024-03-29T11:39:42.046 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:42.046 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:42.046 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:39:42.045112+0000 2024-03-29T11:39:42.046 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:39:42.045112+0000 2024-03-29T11:39:42.046 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:42.046 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:42.046 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:42.046 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:39:42.046 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:39:42.046 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:42.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.045+0000 7f4844693640 1 Processor -- start 2024-03-29T11:39:42.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.045+0000 7f4844693640 1 -- start start 2024-03-29T11:39:42.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.045+0000 7f4844693640 10 monclient: init 2024-03-29T11:39:42.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.046+0000 7f4844693640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:42.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.046+0000 7f4844693640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:42.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.046+0000 7f4844693640 10 monclient(hunting): picked mon.noname-a con 0x7f483c1422b0 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:42.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f4844693640 10 monclient(hunting): picked mon.noname-c con 0x7f483c142690 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:42.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f4844693640 10 monclient(hunting): picked mon.noname-b con 0x7f483c0b17d0 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:42.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f4844693640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c142c80 con 0x7f483c1422b0 2024-03-29T11:39:42.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f4844693640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c0aae40 con 0x7f483c142690 2024-03-29T11:39:42.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f4844693640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c13fbd0 con 0x7f483c0b17d0 2024-03-29T11:39:42.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f4844693640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:42.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f4844693640 10 monclient(hunting): authenticate will time out at 1161.496275s 2024-03-29T11:39:42.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f4841c2e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f483c142690 0x7f483c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:49260/0 (socket says 172.21.15.37:49260) 2024-03-29T11:39:42.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f4841c2e640 1 -- 172.21.15.37:0/3470331974 learned_addr learned my addr 172.21.15.37:0/3470331974 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:42.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f484142d640 1 -- 172.21.15.37:0/3470331974 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092627589 0 0) 0x7f483c0aae40 con 0x7f483c142690 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f484142d640 10 monclient(hunting): _negotiate 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f484142d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f484142d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f484142d640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.047+0000 7f484142d640 1 -- 172.21.15.37:0/3470331974 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4828003520 con 0x7f483c142690 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.048+0000 7f484142d640 1 -- 172.21.15.37:0/3470331974 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059504709 0 0) 0x7f483c13fbd0 con 0x7f483c0b17d0 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.048+0000 7f484142d640 10 monclient(hunting): _negotiate 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.048+0000 7f484142d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.048+0000 7f484142d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.048+0000 7f484142d640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.048+0000 7f484142d640 1 -- 172.21.15.37:0/3470331974 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f483c0aae40 con 0x7f483c0b17d0 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.048+0000 7f484142d640 1 -- 172.21.15.37:0/3470331974 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2857399731 0 0) 0x7f483c0aae40 con 0x7f483c0b17d0 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.048+0000 7f484142d640 1 -- 172.21.15.37:0/3470331974 >> v1:172.21.15.146:6789/0 conn(0x7f483c142690 legacy=0x7f483c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.048+0000 7f484142d640 1 -- 172.21.15.37:0/3470331974 >> v1:172.21.15.37:6789/0 conn(0x7f483c1422b0 legacy=0x7f483c13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.049+0000 7f484142d640 10 monclient: _finish_hunting 0 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.049+0000 7f484142d640 1 monclient: found mon.noname-b 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.049+0000 7f484142d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.049+0000 7f484142d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.049+0000 7f484142d640 1 -- 172.21.15.37:0/3470331974 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c151940 con 0x7f483c0b17d0 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.049+0000 7f484142d640 10 monclient: _finish_auth 0 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.049+0000 7f484142d640 10 monclient: _check_auth_tickets 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.049+0000 7f484142d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.049+0000 7f484142d640 1 -- 172.21.15.37:0/3470331974 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4834001670 con 0x7f483c0b17d0 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.049+0000 7f484142d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.049+0000 7f484142d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.049+0000 7f484142d640 10 monclient: dump: 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.049+0000 7f4844693640 5 monclient: authenticate success, global_id 4418 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.049+0000 7f4844693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.050+0000 7f484142d640 1 -- 172.21.15.37:0/3470331974 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48340017d0 con 0x7f483c0b17d0 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.050+0000 7f484142d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.050+0000 7f484142d640 1 -- 172.21.15.37:0/3470331974 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4834001ba0 con 0x7f483c0b17d0 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.050+0000 7f4844693640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.050+0000 7f4844693640 4 set_mon_vals no callback set 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.050+0000 7f484142d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.050+0000 7f484142d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.050+0000 7f484142d640 10 monclient: dump: 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.051+0000 7f4844693640 10 monclient: shutdown 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.051+0000 7f4844693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.051+0000 7f4844693640 1 -- 172.21.15.37:0/3470331974 >> v1:172.21.15.37:6790/0 conn(0x7f483c0b17d0 legacy=0x7f483c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.051+0000 7f4844693640 1 -- 172.21.15.37:0/3470331974 shutdown_connections 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.051+0000 7f4844693640 1 -- 172.21.15.37:0/3470331974 >> 172.21.15.37:0/3470331974 conn(0x7f483c066a00 msgr2=0x7f483c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.051+0000 7f4844693640 1 -- 172.21.15.37:0/3470331974 shutdown_connections 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.051+0000 7f4844693640 1 -- 172.21.15.37:0/3470331974 wait complete. 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.052+0000 7f4844693640 10 monclient: build_initial_monmap 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.052+0000 7f4844693640 10 monclient: monmap: 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.052+0000 7f4844693640 1 librados: starting msgr at 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.052+0000 7f4844693640 1 librados: starting objecter 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.052+0000 7f4844693640 1 Processor -- start 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.052+0000 7f4844693640 1 -- start start 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.052+0000 7f4844693640 1 librados: setting wanted keys 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.052+0000 7f4844693640 1 librados: calling monclient init 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.052+0000 7f4844693640 10 monclient: init 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.053+0000 7f4844693640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.053+0000 7f4844693640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.053+0000 7f4844693640 10 monclient(hunting): picked mon.noname-b con 0x7f483c0b17d0 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.053+0000 7f4844693640 10 monclient(hunting): picked mon.noname-c con 0x7f483c1422b0 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.053+0000 7f4844693640 10 monclient(hunting): picked mon.noname-a con 0x7f483c142690 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.053+0000 7f4844693640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c150d70 con 0x7f483c142690 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f4844693640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c0ac910 con 0x7f483c0b17d0 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f4844693640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c0acbf0 con 0x7f483c1422b0 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f4841c2e640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f483c1422b0 0x7f483c0ab820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:33638/0 (socket says 172.21.15.37:33638) 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f4841c2e640 1 -- 172.21.15.37:0/629541491 learned_addr learned my addr 172.21.15.37:0/629541491 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f4844693640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f4844693640 10 monclient(hunting): authenticate will time out at 1161.503201s 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 10 client.?.objecter ms_handle_connect 0x7f483c0b17d0 2024-03-29T11:39:42.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:42.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687646222 0 0) 0x7f483c0ac910 con 0x7f483c0b17d0 2024-03-29T11:39:42.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:42.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:42.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:42.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:39:42.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4820004eb0 con 0x7f483c0b17d0 2024-03-29T11:39:42.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 10 client.?.objecter ms_handle_connect 0x7f483c1422b0 2024-03-29T11:39:42.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:42.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1356270624 0 0) 0x7f4820004eb0 con 0x7f483c0b17d0 2024-03-29T11:39:42.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f483c0ac910 con 0x7f483c0b17d0 2024-03-29T11:39:42.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4830001bf0 con 0x7f483c0b17d0 2024-03-29T11:39:42.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:42.067 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 10 monclient(hunting): dump: 2024-03-29T11:39:42.067 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:42.067 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:42.067 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:42.067 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:42.067 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.067 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:42.067 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:42.067 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:42.067 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:42.067 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:42.067 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.054+0000 7f482effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:42.067 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.055+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 117768937 0 0) 0x7f483c0ac910 con 0x7f483c0b17d0 2024-03-29T11:39:42.067 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.055+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 >> v1:172.21.15.37:6790/0 conn(0x7f483c1422b0 legacy=0x7f483c0ab820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.055+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 >> v1:172.21.15.37:6789/0 conn(0x7f483c142690 legacy=0x7f483c1e1c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f482effd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f482effd640 1 monclient: found mon.b 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f482effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f482effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c0aced0 con 0x7f483c0b17d0 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f482effd640 10 monclient: _finish_auth 0 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f482effd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f482effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f4844693640 5 monclient: authenticate success, global_id 4366 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f4844693640 10 monclient: _renew_subs 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f4844693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f4844693640 1 -- 172.21.15.37:0/629541491 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c150a00 con 0x7f483c0b17d0 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f4844693640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f4844693640 10 monclient: _renew_subs 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f4844693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f4844693640 1 -- 172.21.15.37:0/629541491 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c0b6460 con 0x7f483c0b17d0 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f4844693640 1 librados: init done 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4830003970 con 0x7f483c0b17d0 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f482effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4830003c80 con 0x7f483c0b17d0 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f482effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f482effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f4840c2c640 4 set_mon_vals no callback set 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.057+0000 7f482effd640 10 monclient: dump: 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.059+0000 7f4844693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.059+0000 7f4844693640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.059+0000 7f4844693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.059+0000 7f4844693640 1 -- 172.21.15.37:0/629541491 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483c075f10 con 0x7f483c0b17d0 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.060+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f4830019c50 con 0x7f483c0b17d0 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.060+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f4830083bf0 con 0x7f483c0b17d0 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.060+0000 7f482effd640 10 client.4366.objecter ms_dispatch 0x7f483c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.060+0000 7f482effd640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.060+0000 7f482effd640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.061+0000 7f482effd640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.062+0000 7f4841c2e640 10 monclient: get_auth_request con 0x7f482006dcb0 auth_method 0 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.063+0000 7f482effd640 10 client.4366.objecter ms_handle_connect 0x7f482006dcb0 2024-03-29T11:39:42.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.068+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f483004f4d0 con 0x7f483c0b17d0 2024-03-29T11:39:42.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.068+0000 7f482effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:42.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.068+0000 7f482effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:42.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.381+0000 7f4844693640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:42.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.381+0000 7f4844693640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:42.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.381+0000 7f4844693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:39:42.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.381+0000 7f4844693640 1 -- 172.21.15.37:0/629541491 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f483c0b2270 con 0x7f483c0b17d0 2024-03-29T11:39:42.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.382+0000 7f482effd640 1 -- 172.21.15.37:0/629541491 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f4830051400 con 0x7f483c0b17d0 2024-03-29T11:39:42.383 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:39:42.383 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:42.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.382+0000 7f482effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:42.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.382+0000 7f482effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:42.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.385+0000 7f4844693640 10 librados: watch_flush enter 2024-03-29T11:39:42.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.385+0000 7f4844693640 10 librados: watch_flush exit 2024-03-29T11:39:42.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.385+0000 7f4844693640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-29T11:39:42.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.385+0000 7f4844693640 10 client.4366.objecter successfully canceled tick 2024-03-29T11:39:42.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.385+0000 7f4844693640 1 -- 172.21.15.37:0/629541491 >> v1:172.21.15.146:6816/2899149139 conn(0x7f482006dcb0 legacy=0x7f4820070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.385+0000 7f4844693640 10 monclient: shutdown 2024-03-29T11:39:42.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.385+0000 7f4844693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:42.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.385+0000 7f4844693640 1 -- 172.21.15.37:0/629541491 >> v1:172.21.15.146:6789/0 conn(0x7f483c0b17d0 legacy=0x7f483c12e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.385+0000 7f4844693640 1 -- 172.21.15.37:0/629541491 shutdown_connections 2024-03-29T11:39:42.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.385+0000 7f4844693640 1 -- 172.21.15.37:0/629541491 >> 172.21.15.37:0/629541491 conn(0x7f483c066a00 msgr2=0x7f483c0b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:42.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.385+0000 7f4844693640 1 -- 172.21.15.37:0/629541491 shutdown_connections 2024-03-29T11:39:42.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.385+0000 7f4844693640 1 -- 172.21.15.37:0/629541491 wait complete. 2024-03-29T11:39:42.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:42.385+0000 7f4844693640 1 librados: shutdown 2024-03-29T11:39:42.400 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:49.207 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:49.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.351+0000 7f2a045e2640 10 monclient: get_monmap_and_config 2024-03-29T11:39:49.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.351+0000 7f2a045e2640 10 monclient: build_initial_monmap 2024-03-29T11:39:49.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.351+0000 7f2a045e2640 10 monclient: monmap: 2024-03-29T11:39:49.352 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:49.352 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:49.352 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:39:49.352035+0000 2024-03-29T11:39:49.352 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:39:49.352035+0000 2024-03-29T11:39:49.353 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:49.353 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:49.353 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:49.353 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:39:49.353 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:39:49.353 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:49.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.353+0000 7f2a045e2640 1 Processor -- start 2024-03-29T11:39:49.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.353+0000 7f2a045e2640 1 -- start start 2024-03-29T11:39:49.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.353+0000 7f2a045e2640 10 monclient: init 2024-03-29T11:39:49.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.354+0000 7f2a045e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:49.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.354+0000 7f2a045e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:49.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.354+0000 7f2a045e2640 10 monclient(hunting): picked mon.noname-b con 0x7f29fc1457f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:49.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a045e2640 10 monclient(hunting): picked mon.noname-a con 0x7f29fc13f650 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:49.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a045e2640 10 monclient(hunting): picked mon.noname-c con 0x7f29fc0a9d70 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:49.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a045e2640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29fc0aaef0 con 0x7f29fc13f650 2024-03-29T11:39:49.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a045e2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29fc145bd0 con 0x7f29fc0a9d70 2024-03-29T11:39:49.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a045e2640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29fc145eb0 con 0x7f29fc1457f0 2024-03-29T11:39:49.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a045e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:49.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a045e2640 10 monclient(hunting): authenticate will time out at 1168.804316s 2024-03-29T11:39:49.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a0237e640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f29fc1457f0 0x7f29fc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:43410/0 (socket says 172.21.15.37:43410) 2024-03-29T11:39:49.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a0237e640 1 -- 172.21.15.37:0/960998701 learned_addr learned my addr 172.21.15.37:0/960998701 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:49.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a0137c640 1 -- 172.21.15.37:0/960998701 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236885169 0 0) 0x7f29fc145eb0 con 0x7f29fc1457f0 2024-03-29T11:39:49.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a0137c640 10 monclient(hunting): _negotiate 2024-03-29T11:39:49.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a0137c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:49.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a0137c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:49.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a0137c640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:39:49.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.355+0000 7f2a0137c640 1 -- 172.21.15.37:0/960998701 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e8003520 con 0x7f29fc1457f0 2024-03-29T11:39:49.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.356+0000 7f2a0137c640 1 -- 172.21.15.37:0/960998701 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1663176738 0 0) 0x7f29e8003520 con 0x7f29fc1457f0 2024-03-29T11:39:49.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.356+0000 7f2a0137c640 1 -- 172.21.15.37:0/960998701 >> v1:172.21.15.146:6789/0 conn(0x7f29fc0a9d70 legacy=0x7f29fc14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:49.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.358+0000 7f2a0137c640 1 -- 172.21.15.37:0/960998701 >> v1:172.21.15.37:6789/0 conn(0x7f29fc13f650 legacy=0x7f29fc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:49.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.358+0000 7f2a0137c640 10 monclient: _finish_hunting 0 2024-03-29T11:39:49.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.358+0000 7f2a0137c640 1 monclient: found mon.noname-b 2024-03-29T11:39:49.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.358+0000 7f2a0137c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:49.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.358+0000 7f2a0137c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:39:49.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.358+0000 7f2a0137c640 1 -- 172.21.15.37:0/960998701 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc146340 con 0x7f29fc1457f0 2024-03-29T11:39:49.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.358+0000 7f2a0137c640 10 monclient: _finish_auth 0 2024-03-29T11:39:49.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.358+0000 7f2a0137c640 10 monclient: _check_auth_tickets 2024-03-29T11:39:49.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.359+0000 7f2a0137c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:49.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.359+0000 7f2a0137c640 1 -- 172.21.15.37:0/960998701 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f29ec001690 con 0x7f29fc1457f0 2024-03-29T11:39:49.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.359+0000 7f2a0137c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:49.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.359+0000 7f2a0137c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:49.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.359+0000 7f2a0137c640 10 monclient: dump: 2024-03-29T11:39:49.362 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:49.362 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:49.362 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:49.362 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:49.362 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:49.362 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:49.362 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:49.362 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:49.362 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:49.362 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:49.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.359+0000 7f2a045e2640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:39:49.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.359+0000 7f2a045e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:49.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.359+0000 7f2a0137c640 1 -- 172.21.15.37:0/960998701 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29ec001920 con 0x7f29fc1457f0 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.359+0000 7f2a0137c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.359+0000 7f2a0137c640 1 -- 172.21.15.37:0/960998701 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f29ec001cb0 con 0x7f29fc1457f0 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.359+0000 7f2a045e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.359+0000 7f2a045e2640 4 set_mon_vals no callback set 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.360+0000 7f2a0137c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.360+0000 7f2a0137c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.360+0000 7f2a0137c640 10 monclient: dump: 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:49.363 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:49.364 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:49.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.360+0000 7f2a045e2640 10 monclient: shutdown 2024-03-29T11:39:49.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.360+0000 7f2a045e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:49.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.360+0000 7f2a045e2640 1 -- 172.21.15.37:0/960998701 >> v1:172.21.15.37:6790/0 conn(0x7f29fc1457f0 legacy=0x7f29fc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:49.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.361+0000 7f2a045e2640 1 -- 172.21.15.37:0/960998701 shutdown_connections 2024-03-29T11:39:49.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.361+0000 7f2a045e2640 1 -- 172.21.15.37:0/960998701 >> 172.21.15.37:0/960998701 conn(0x7f29fc066a00 msgr2=0x7f29fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:49.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.362+0000 7f2a045e2640 1 -- 172.21.15.37:0/960998701 shutdown_connections 2024-03-29T11:39:49.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.362+0000 7f2a045e2640 1 -- 172.21.15.37:0/960998701 wait complete. 2024-03-29T11:39:49.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.363+0000 7f2a045e2640 10 monclient: build_initial_monmap 2024-03-29T11:39:49.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.363+0000 7f2a045e2640 10 monclient: monmap: 2024-03-29T11:39:49.366 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:49.367 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:49.367 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:39:49.367 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:39:49.367 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:49.367 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:49.367 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:49.367 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:39:49.367 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:39:49.367 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:49.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.364+0000 7f2a045e2640 1 librados: starting msgr at 2024-03-29T11:39:49.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.364+0000 7f2a045e2640 1 librados: starting objecter 2024-03-29T11:39:49.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.364+0000 7f2a045e2640 1 Processor -- start 2024-03-29T11:39:49.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.364+0000 7f2a045e2640 1 -- start start 2024-03-29T11:39:49.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.364+0000 7f2a045e2640 1 librados: setting wanted keys 2024-03-29T11:39:49.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.364+0000 7f2a045e2640 1 librados: calling monclient init 2024-03-29T11:39:49.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.364+0000 7f2a045e2640 10 monclient: init 2024-03-29T11:39:49.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.365+0000 7f2a045e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:49.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.365+0000 7f2a045e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:49.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.365+0000 7f2a045e2640 10 monclient(hunting): picked mon.noname-a con 0x7f29fc0a9d70 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:49.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.365+0000 7f2a045e2640 10 monclient(hunting): picked mon.noname-b con 0x7f29fc13f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:49.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.365+0000 7f2a045e2640 10 monclient(hunting): picked mon.noname-c con 0x7f29fc1457f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:49.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.365+0000 7f2a02b7f640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f29fc1457f0 0x7f29fc1deb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:44688/0 (socket says 172.21.15.37:44688) 2024-03-29T11:39:49.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.366+0000 7f2a02b7f640 1 -- 172.21.15.37:0/1115916022 learned_addr learned my addr 172.21.15.37:0/1115916022 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:49.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.366+0000 7f2a045e2640 1 -- 172.21.15.37:0/1115916022 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29fc135750 con 0x7f29fc0a9d70 2024-03-29T11:39:49.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.366+0000 7f29f2ffd640 10 client.?.objecter ms_handle_connect 0x7f29fc1457f0 2024-03-29T11:39:49.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.366+0000 7f29f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:49.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.366+0000 7f2a045e2640 1 -- 172.21.15.37:0/1115916022 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29fc1359a0 con 0x7f29fc13f650 2024-03-29T11:39:49.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.366+0000 7f2a045e2640 1 -- 172.21.15.37:0/1115916022 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29fc1417e0 con 0x7f29fc1457f0 2024-03-29T11:39:49.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.366+0000 7f2a045e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:49.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.366+0000 7f2a045e2640 10 monclient(hunting): authenticate will time out at 1168.815582s 2024-03-29T11:39:49.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.369+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165182093 0 0) 0x7f29fc1417e0 con 0x7f29fc1457f0 2024-03-29T11:39:49.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.369+0000 7f29f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:49.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.369+0000 7f29f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:49.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.369+0000 7f29f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:49.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.369+0000 7f29f2ffd640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:39:49.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.369+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e0004eb0 con 0x7f29fc1457f0 2024-03-29T11:39:49.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.369+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2243333606 0 0) 0x7f29e0004eb0 con 0x7f29fc1457f0 2024-03-29T11:39:49.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29fc1417e0 con 0x7f29fc1457f0 2024-03-29T11:39:49.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f29f4001b70 con 0x7f29fc1457f0 2024-03-29T11:39:49.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:49.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:49.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 10 monclient(hunting): dump: 2024-03-29T11:39:49.374 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:49.375 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:49.375 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:49.375 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:49.375 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:49.375 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:49.375 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:49.375 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:49.375 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:49.375 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:49.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:49.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3554776362 0 0) 0x7f29fc1417e0 con 0x7f29fc1457f0 2024-03-29T11:39:49.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 >> v1:172.21.15.146:6789/0 conn(0x7f29fc13f650 legacy=0x7f29fc1db720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:49.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 >> v1:172.21.15.37:6789/0 conn(0x7f29fc0a9d70 legacy=0x7f29fc13b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:49.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 1 monclient: found mon.c 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc13fda0 con 0x7f29fc1457f0 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.370+0000 7f29f2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f29f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f29f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29f4001cd0 con 0x7f29fc1457f0 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f2a045e2640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f2a045e2640 10 monclient: _renew_subs 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f2a045e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f2a045e2640 1 -- 172.21.15.37:0/1115916022 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29fc1dbe30 con 0x7f29fc1457f0 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f2a045e2640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f2a045e2640 10 monclient: _renew_subs 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f2a045e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f2a045e2640 1 -- 172.21.15.37:0/1115916022 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29fc1df280 con 0x7f29fc1457f0 2024-03-29T11:39:49.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f2a045e2640 1 librados: init done 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f29f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f29f4004e20 con 0x7f29fc1457f0 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.371+0000 7f29f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.372+0000 7f29f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.372+0000 7f29f2ffd640 10 monclient: dump: 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:49.377 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:49.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.372+0000 7f2a00b7b640 4 set_mon_vals no callback set 2024-03-29T11:39:49.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.373+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f29f401adf0 con 0x7f29fc1457f0 2024-03-29T11:39:49.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.374+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f29f4084e50 con 0x7f29fc1457f0 2024-03-29T11:39:49.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.374+0000 7f29f2ffd640 10 client.4424.objecter ms_dispatch 0x7f29fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:49.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.374+0000 7f29f2ffd640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:49.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.374+0000 7f29f2ffd640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:49.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.374+0000 7f2a045e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:49.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.374+0000 7f2a045e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:49.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.374+0000 7f2a045e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:49.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.374+0000 7f29f2ffd640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T11:39:49.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.374+0000 7f2a045e2640 1 -- 172.21.15.37:0/1115916022 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e4005080 con 0x7f29fc1457f0 2024-03-29T11:39:49.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.374+0000 7f2a0237e640 10 monclient: get_auth_request con 0x7f29e006dc00 auth_method 0 2024-03-29T11:39:49.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.375+0000 7f29f2ffd640 10 client.4424.objecter ms_handle_connect 0x7f29e006dc00 2024-03-29T11:39:49.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.384+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29f4050560 con 0x7f29fc1457f0 2024-03-29T11:39:49.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.384+0000 7f29f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:49.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.384+0000 7f29f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:49.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.708+0000 7f2a045e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:49.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.708+0000 7f2a045e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:49.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.708+0000 7f2a045e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:49.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.708+0000 7f2a045e2640 1 -- 172.21.15.37:0/1115916022 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29e4005330 con 0x7f29fc1457f0 2024-03-29T11:39:49.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.710+0000 7f29f2ffd640 1 -- 172.21.15.37:0/1115916022 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f29f4052660 con 0x7f29fc1457f0 2024-03-29T11:39:49.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.710+0000 7f29f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:49.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.710+0000 7f29f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:49.711 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:39:49.711 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:49.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.713+0000 7f2a045e2640 10 librados: watch_flush enter 2024-03-29T11:39:49.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.713+0000 7f2a045e2640 10 librados: watch_flush exit 2024-03-29T11:39:49.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.713+0000 7f2a045e2640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T11:39:49.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.713+0000 7f2a045e2640 10 client.4424.objecter successfully canceled tick 2024-03-29T11:39:49.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.713+0000 7f2a045e2640 1 -- 172.21.15.37:0/1115916022 >> v1:172.21.15.146:6816/2899149139 conn(0x7f29e006dc00 legacy=0x7f29e00700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:49.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.713+0000 7f2a045e2640 10 monclient: shutdown 2024-03-29T11:39:49.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.713+0000 7f2a045e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:49.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.713+0000 7f2a045e2640 1 -- 172.21.15.37:0/1115916022 >> v1:172.21.15.37:6790/0 conn(0x7f29fc1457f0 legacy=0x7f29fc1deb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:49.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.714+0000 7f2a045e2640 1 -- 172.21.15.37:0/1115916022 shutdown_connections 2024-03-29T11:39:49.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.714+0000 7f2a045e2640 1 -- 172.21.15.37:0/1115916022 >> 172.21.15.37:0/1115916022 conn(0x7f29fc066a00 msgr2=0x7f29fc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:49.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.714+0000 7f2a045e2640 1 -- 172.21.15.37:0/1115916022 shutdown_connections 2024-03-29T11:39:49.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.714+0000 7f2a045e2640 1 -- 172.21.15.37:0/1115916022 wait complete. 2024-03-29T11:39:49.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:49.714+0000 7f2a045e2640 1 librados: shutdown 2024-03-29T11:39:49.732 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:56.538 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:56.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.685+0000 7f47b123b640 10 monclient: get_monmap_and_config 2024-03-29T11:39:56.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.685+0000 7f47b123b640 10 monclient: build_initial_monmap 2024-03-29T11:39:56.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.685+0000 7f47b123b640 10 monclient: monmap: 2024-03-29T11:39:56.688 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:56.688 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:56.688 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:39:56.686799+0000 2024-03-29T11:39:56.688 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:39:56.686799+0000 2024-03-29T11:39:56.688 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:56.688 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:56.688 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:56.688 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:39:56.688 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:39:56.688 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:56.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.687+0000 7f47b123b640 1 Processor -- start 2024-03-29T11:39:56.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.687+0000 7f47b123b640 1 -- start start 2024-03-29T11:39:56.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.687+0000 7f47b123b640 10 monclient: init 2024-03-29T11:39:56.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.688+0000 7f47b123b640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:56.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.688+0000 7f47b123b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:56.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.688+0000 7f47b123b640 10 monclient(hunting): picked mon.noname-a con 0x7f47ac14a400 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:56.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.688+0000 7f47b123b640 10 monclient(hunting): picked mon.noname-b con 0x7f47ac0aa500 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:56.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.688+0000 7f47b123b640 10 monclient(hunting): picked mon.noname-c con 0x7f47ac149800 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:56.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.691+0000 7f47b123b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac0aca50 con 0x7f47ac14a400 2024-03-29T11:39:56.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.689+0000 7f47ab7fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f47ac0aa500 0x7f47ac0ad160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:39840/0 (socket says 172.21.15.37:39840) 2024-03-29T11:39:56.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.691+0000 7f47ab7fe640 1 -- 172.21.15.37:0/444149505 learned_addr learned my addr 172.21.15.37:0/444149505 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:56.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.691+0000 7f47b123b640 1 -- 172.21.15.37:0/444149505 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac149cb0 con 0x7f47ac149800 2024-03-29T11:39:56.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.691+0000 7f47b123b640 1 -- 172.21.15.37:0/444149505 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac0acc80 con 0x7f47ac0aa500 2024-03-29T11:39:56.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.691+0000 7f47b123b640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:56.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.691+0000 7f47b123b640 10 monclient(hunting): authenticate will time out at 1176.140223s 2024-03-29T11:39:56.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 1 -- 172.21.15.37:0/444149505 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691145093 0 0) 0x7f47ac0acc80 con 0x7f47ac0aa500 2024-03-29T11:39:56.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:56.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:56.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:56.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:39:56.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 1 -- 172.21.15.37:0/444149505 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4794003520 con 0x7f47ac0aa500 2024-03-29T11:39:56.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 1 -- 172.21.15.37:0/444149505 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3075170779 0 0) 0x7f4794003520 con 0x7f47ac0aa500 2024-03-29T11:39:56.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 1 -- 172.21.15.37:0/444149505 >> v1:172.21.15.146:6789/0 conn(0x7f47ac149800 legacy=0x7f47ac151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:56.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 1 -- 172.21.15.37:0/444149505 >> v1:172.21.15.37:6789/0 conn(0x7f47ac14a400 legacy=0x7f47ac0aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:56.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:56.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 1 monclient: found mon.noname-b 2024-03-29T11:39:56.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:56.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:39:56.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 1 -- 172.21.15.37:0/444149505 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac0ac350 con 0x7f47ac0aa500 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.693+0000 7f47aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.694+0000 7f47aaffd640 1 -- 172.21.15.37:0/444149505 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f47a0001690 con 0x7f47ac0aa500 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.694+0000 7f47aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.694+0000 7f47aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.694+0000 7f47aaffd640 10 monclient: dump: 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:56.697 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.694+0000 7f47b123b640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.694+0000 7f47b123b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.694+0000 7f47aaffd640 1 -- 172.21.15.37:0/444149505 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47a0001860 con 0x7f47ac0aa500 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.694+0000 7f47aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.694+0000 7f47aaffd640 1 -- 172.21.15.37:0/444149505 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f47a0001cb0 con 0x7f47ac0aa500 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.694+0000 7f47aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.694+0000 7f47aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.694+0000 7f47aaffd640 10 monclient: dump: 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:56.698 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:56.699 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:56.699 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:56.699 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:56.699 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:56.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.694+0000 7f47b123b640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:56.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.694+0000 7f47b123b640 4 set_mon_vals no callback set 2024-03-29T11:39:56.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.695+0000 7f47b123b640 10 monclient: shutdown 2024-03-29T11:39:56.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.695+0000 7f47b123b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:56.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.695+0000 7f47b123b640 1 -- 172.21.15.37:0/444149505 >> v1:172.21.15.37:6790/0 conn(0x7f47ac0aa500 legacy=0x7f47ac0ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.696+0000 7f47b123b640 1 -- 172.21.15.37:0/444149505 shutdown_connections 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.696+0000 7f47b123b640 1 -- 172.21.15.37:0/444149505 >> 172.21.15.37:0/444149505 conn(0x7f47ac0b3a30 msgr2=0x7f47ac0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.696+0000 7f47b123b640 1 -- 172.21.15.37:0/444149505 shutdown_connections 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.696+0000 7f47b123b640 1 -- 172.21.15.37:0/444149505 wait complete. 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.697+0000 7f47b123b640 10 monclient: build_initial_monmap 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.697+0000 7f47b123b640 10 monclient: monmap: 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:39:56.701 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:56.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.698+0000 7f47b123b640 1 librados: starting msgr at 2024-03-29T11:39:56.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.698+0000 7f47b123b640 1 librados: starting objecter 2024-03-29T11:39:56.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.698+0000 7f47b123b640 1 Processor -- start 2024-03-29T11:39:56.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.698+0000 7f47b123b640 1 -- start start 2024-03-29T11:39:56.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.698+0000 7f47b123b640 1 librados: setting wanted keys 2024-03-29T11:39:56.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.699+0000 7f47b123b640 1 librados: calling monclient init 2024-03-29T11:39:56.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.699+0000 7f47b123b640 10 monclient: init 2024-03-29T11:39:56.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.700+0000 7f47b123b640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:56.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.700+0000 7f47b123b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:56.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.700+0000 7f47b123b640 10 monclient(hunting): picked mon.noname-a con 0x7f47ac0aa500 addr v1:172.21.15.37:6789/0 2024-03-29T11:39:56.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.701+0000 7f47b123b640 10 monclient(hunting): picked mon.noname-c con 0x7f47ac149800 addr v1:172.21.15.37:6790/0 2024-03-29T11:39:56.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.701+0000 7f47b123b640 10 monclient(hunting): picked mon.noname-b con 0x7f47ac14a400 addr v1:172.21.15.146:6789/0 2024-03-29T11:39:56.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.701+0000 7f47b123b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac1488b0 con 0x7f47ac0aa500 2024-03-29T11:39:56.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.701+0000 7f47b123b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac13efe0 con 0x7f47ac14a400 2024-03-29T11:39:56.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.701+0000 7f47b123b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac146a80 con 0x7f47ac149800 2024-03-29T11:39:56.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.701+0000 7f47b123b640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:56.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.701+0000 7f47b123b640 10 monclient(hunting): authenticate will time out at 1176.150901s 2024-03-29T11:39:56.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.704+0000 7f47abfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f47ac0aa500 0x7f47ac1474f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:60083/0 (socket says 172.21.15.37:60083) 2024-03-29T11:39:56.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.704+0000 7f47abfff640 1 -- 172.21.15.37:0/536473665 learned_addr learned my addr 172.21.15.37:0/536473665 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:39:56.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 client.?.objecter ms_handle_connect 0x7f47ac0aa500 2024-03-29T11:39:56.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:56.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 client.?.objecter ms_handle_connect 0x7f47ac14a400 2024-03-29T11:39:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 client.?.objecter ms_handle_connect 0x7f47ac149800 2024-03-29T11:39:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689685551 0 0) 0x7f47ac1488b0 con 0x7f47ac0aa500 2024-03-29T11:39:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:39:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:39:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4798004eb0 con 0x7f47ac0aa500 2024-03-29T11:39:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610634524 0 0) 0x7f47ac146a80 con 0x7f47ac149800 2024-03-29T11:39:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:39:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:56.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:39:56.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47ac1488b0 con 0x7f47ac149800 2024-03-29T11:39:56.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826066507 0 0) 0x7f47ac13efe0 con 0x7f47ac14a400 2024-03-29T11:39:56.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:39:56.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:56.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.707+0000 7f47a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:56.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.708+0000 7f47a8ff9640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:39:56.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.708+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47ac146a80 con 0x7f47ac14a400 2024-03-29T11:39:56.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.708+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1141009272 0 0) 0x7f47ac1488b0 con 0x7f47ac149800 2024-03-29T11:39:56.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.708+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47ac13efe0 con 0x7f47ac149800 2024-03-29T11:39:56.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.708+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1753444468 0 0) 0x7f47ac146a80 con 0x7f47ac14a400 2024-03-29T11:39:56.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.708+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47ac1488b0 con 0x7f47ac14a400 2024-03-29T11:39:56.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.708+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1883236777 0 0) 0x7f4798004eb0 con 0x7f47ac0aa500 2024-03-29T11:39:56.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.708+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47ac146a80 con 0x7f47ac0aa500 2024-03-29T11:39:56.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.708+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f47a0001b70 con 0x7f47ac149800 2024-03-29T11:39:56.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.708+0000 7f47a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:56.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.708+0000 7f47a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:56.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.708+0000 7f47a8ff9640 10 monclient(hunting): dump: 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.708+0000 7f47a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.708+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f47a40019e0 con 0x7f47ac14a400 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:56.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 10 monclient(hunting): dump: 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f479c001750 con 0x7f47ac0aa500 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 10 monclient(hunting): dump: 2024-03-29T11:39:56.717 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3612075762 0 0) 0x7f47ac13efe0 con 0x7f47ac149800 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 >> v1:172.21.15.146:6789/0 conn(0x7f47ac14a400 legacy=0x7f47ac144b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 >> v1:172.21.15.37:6789/0 conn(0x7f47ac0aa500 legacy=0x7f47ac1474f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 1 monclient: found mon.c 2024-03-29T11:39:56.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac147c00 con 0x7f47ac149800 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.709+0000 7f47a8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.710+0000 7f47a8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.710+0000 7f47a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.710+0000 7f47b123b640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.710+0000 7f47b123b640 10 monclient: _renew_subs 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.710+0000 7f47b123b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.710+0000 7f47b123b640 1 -- 172.21.15.37:0/536473665 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ac1e6d70 con 0x7f47ac149800 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.710+0000 7f47b123b640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.710+0000 7f47b123b640 10 monclient: _renew_subs 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.710+0000 7f47b123b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.710+0000 7f47b123b640 1 -- 172.21.15.37:0/536473665 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ac1e70e0 con 0x7f47ac149800 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.710+0000 7f47b123b640 1 librados: init done 2024-03-29T11:39:56.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.711+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47a0001010 con 0x7f47ac149800 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.712+0000 7f47a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.712+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f47a0004bf0 con 0x7f47ac149800 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.712+0000 7f47a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.712+0000 7f47a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.712+0000 7f47a8ff9640 10 monclient: dump: 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:39:56.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.712+0000 7f47aa7fc640 4 set_mon_vals no callback set 2024-03-29T11:39:56.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.712+0000 7f47b123b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:56.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.713+0000 7f47b123b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:56.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.713+0000 7f47b123b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:56.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.713+0000 7f47b123b640 1 -- 172.21.15.37:0/536473665 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478c005080 con 0x7f47ac149800 2024-03-29T11:39:56.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.714+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f47a001abc0 con 0x7f47ac149800 2024-03-29T11:39:56.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.715+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f47a00849b0 con 0x7f47ac149800 2024-03-29T11:39:56.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.715+0000 7f47a8ff9640 10 client.4430.objecter ms_dispatch 0x7f47ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:56.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.715+0000 7f47a8ff9640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:56.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.715+0000 7f47a8ff9640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:56.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.715+0000 7f47a8ff9640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T11:39:56.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.715+0000 7f47abfff640 10 monclient: get_auth_request con 0x7f479806e170 auth_method 0 2024-03-29T11:39:56.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.715+0000 7f47a8ff9640 10 client.4430.objecter ms_handle_connect 0x7f479806e170 2024-03-29T11:39:56.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.724+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47a0050290 con 0x7f47ac149800 2024-03-29T11:39:56.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.724+0000 7f47a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:56.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:56.724+0000 7f47a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:57.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.055+0000 7f47b123b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:57.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.055+0000 7f47b123b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:57.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.055+0000 7f47b123b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:39:57.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.055+0000 7f47b123b640 1 -- 172.21.15.37:0/536473665 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f478c005330 con 0x7f47ac149800 2024-03-29T11:39:57.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.056+0000 7f47a8ff9640 1 -- 172.21.15.37:0/536473665 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f47a00521c0 con 0x7f47ac149800 2024-03-29T11:39:57.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.056+0000 7f47a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:57.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.056+0000 7f47a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:57.058 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:39:57.058 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:57.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.059+0000 7f47b123b640 10 librados: watch_flush enter 2024-03-29T11:39:57.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.060+0000 7f47b123b640 10 librados: watch_flush exit 2024-03-29T11:39:57.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.060+0000 7f47b123b640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T11:39:57.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.060+0000 7f47b123b640 10 client.4430.objecter successfully canceled tick 2024-03-29T11:39:57.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.060+0000 7f47b123b640 1 -- 172.21.15.37:0/536473665 >> v1:172.21.15.146:6816/2899149139 conn(0x7f479806e170 legacy=0x7f4798070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.060+0000 7f47b123b640 10 monclient: shutdown 2024-03-29T11:39:57.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.060+0000 7f47b123b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:57.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.060+0000 7f47b123b640 1 -- 172.21.15.37:0/536473665 >> v1:172.21.15.37:6790/0 conn(0x7f47ac149800 legacy=0x7f47ac144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.061+0000 7f47b123b640 1 -- 172.21.15.37:0/536473665 shutdown_connections 2024-03-29T11:39:57.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.061+0000 7f47b123b640 1 -- 172.21.15.37:0/536473665 >> 172.21.15.37:0/536473665 conn(0x7f47ac0b3a30 msgr2=0x7f47ac0b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:57.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.061+0000 7f47b123b640 1 -- 172.21.15.37:0/536473665 shutdown_connections 2024-03-29T11:39:57.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.061+0000 7f47b123b640 1 -- 172.21.15.37:0/536473665 wait complete. 2024-03-29T11:39:57.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:39:57.061+0000 7f47b123b640 1 librados: shutdown 2024-03-29T11:39:57.080 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:03.886 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:04.040 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.036+0000 7fa999889640 10 monclient: get_monmap_and_config 2024-03-29T11:40:04.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.036+0000 7fa999889640 10 monclient: build_initial_monmap 2024-03-29T11:40:04.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.036+0000 7fa999889640 10 monclient: monmap: 2024-03-29T11:40:04.041 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:04.041 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:04.041 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:40:04.037519+0000 2024-03-29T11:40:04.041 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:40:04.037519+0000 2024-03-29T11:40:04.041 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:04.041 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:04.041 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:04.041 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:40:04.041 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:40:04.041 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:04.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.038+0000 7fa999889640 1 Processor -- start 2024-03-29T11:40:04.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.038+0000 7fa999889640 1 -- start start 2024-03-29T11:40:04.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.038+0000 7fa999889640 10 monclient: init 2024-03-29T11:40:04.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.039+0000 7fa999889640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:04.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.039+0000 7fa999889640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:04.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.039+0000 7fa999889640 10 monclient(hunting): picked mon.noname-a con 0x7fa98c02b090 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:04.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.039+0000 7fa999889640 10 monclient(hunting): picked mon.noname-b con 0x7fa98c006940 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:04.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.042+0000 7fa999889640 10 monclient(hunting): picked mon.noname-c con 0x7fa98c0eec90 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:04.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.042+0000 7fa999889640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98c006d80 con 0x7fa98c02b090 2024-03-29T11:40:04.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.042+0000 7fa999889640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98c0e2dc0 con 0x7fa98c0eec90 2024-03-29T11:40:04.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.042+0000 7fa999889640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98c0e2f50 con 0x7fa98c006940 2024-03-29T11:40:04.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.042+0000 7fa999889640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:04.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.042+0000 7fa999889640 10 monclient(hunting): authenticate will time out at 1183.491038s 2024-03-29T11:40:04.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.042+0000 7fa998887640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa98c02b090 0x7fa98c0e8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41925/0 (socket says 172.21.15.37:41925) 2024-03-29T11:40:04.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.043+0000 7fa998887640 1 -- 172.21.15.37:0/2874481105 learned_addr learned my addr 172.21.15.37:0/2874481105 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:04.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.046+0000 7fa9937fe640 1 -- 172.21.15.37:0/2874481105 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925473860 0 0) 0x7fa98c006d80 con 0x7fa98c02b090 2024-03-29T11:40:04.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.047+0000 7fa9937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.047+0000 7fa9937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.047+0000 7fa9937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.047+0000 7fa9937fe640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:40:04.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.047+0000 7fa9937fe640 1 -- 172.21.15.37:0/2874481105 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa984003520 con 0x7fa98c02b090 2024-03-29T11:40:04.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.047+0000 7fa9937fe640 1 -- 172.21.15.37:0/2874481105 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 743734530 0 0) 0x7fa984003520 con 0x7fa98c02b090 2024-03-29T11:40:04.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.047+0000 7fa9937fe640 1 -- 172.21.15.37:0/2874481105 >> v1:172.21.15.37:6790/0 conn(0x7fa98c006940 legacy=0x7fa98c029c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.047+0000 7fa9937fe640 1 -- 172.21.15.37:0/2874481105 >> v1:172.21.15.146:6789/0 conn(0x7fa98c0eec90 legacy=0x7fa98c0f1080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:04.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.049+0000 7fa9937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:04.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.049+0000 7fa9937fe640 1 monclient: found mon.noname-a 2024-03-29T11:40:04.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.049+0000 7fa9937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:04.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.049+0000 7fa9937fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:40:04.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.049+0000 7fa9937fe640 1 -- 172.21.15.37:0/2874481105 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c02a430 con 0x7fa98c02b090 2024-03-29T11:40:04.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.049+0000 7fa9937fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:04.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.049+0000 7fa9937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.049+0000 7fa9937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.049+0000 7fa9937fe640 1 -- 172.21.15.37:0/2874481105 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa988001690 con 0x7fa98c02b090 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.049+0000 7fa999889640 5 monclient: authenticate success, global_id 4395 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.050+0000 7fa999889640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.050+0000 7fa9937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.050+0000 7fa9937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.050+0000 7fa9937fe640 10 monclient: dump: 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:04.052 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:04.053 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:04.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.050+0000 7fa999889640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:04.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.050+0000 7fa9937fe640 1 -- 172.21.15.37:0/2874481105 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa988001860 con 0x7fa98c02b090 2024-03-29T11:40:04.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.050+0000 7fa9937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:04.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.050+0000 7fa9937fe640 1 -- 172.21.15.37:0/2874481105 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa988001cb0 con 0x7fa98c02b090 2024-03-29T11:40:04.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.050+0000 7fa9937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.050+0000 7fa9937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:04.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.050+0000 7fa9937fe640 10 monclient: dump: 2024-03-29T11:40:04.055 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:04.055 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:04.055 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:04.055 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:04.056 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.056 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:04.056 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:04.056 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:04.056 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:04.056 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:04.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.050+0000 7fa999889640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:04.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.050+0000 7fa999889640 4 set_mon_vals no callback set 2024-03-29T11:40:04.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.051+0000 7fa999889640 10 monclient: shutdown 2024-03-29T11:40:04.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.051+0000 7fa999889640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:04.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.051+0000 7fa999889640 1 -- 172.21.15.37:0/2874481105 >> v1:172.21.15.37:6789/0 conn(0x7fa98c02b090 legacy=0x7fa98c0e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.054+0000 7fa999889640 1 -- 172.21.15.37:0/2874481105 shutdown_connections 2024-03-29T11:40:04.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.054+0000 7fa999889640 1 -- 172.21.15.37:0/2874481105 >> 172.21.15.37:0/2874481105 conn(0x7fa98c0058f0 msgr2=0x7fa98c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:04.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.054+0000 7fa999889640 1 -- 172.21.15.37:0/2874481105 shutdown_connections 2024-03-29T11:40:04.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.054+0000 7fa999889640 1 -- 172.21.15.37:0/2874481105 wait complete. 2024-03-29T11:40:04.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.055+0000 7fa999889640 10 monclient: build_initial_monmap 2024-03-29T11:40:04.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.055+0000 7fa999889640 10 monclient: monmap: 2024-03-29T11:40:04.058 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:04.058 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:04.058 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:40:04.058 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.055+0000 7fa999889640 1 librados: starting msgr at 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.055+0000 7fa999889640 1 librados: starting objecter 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.055+0000 7fa999889640 1 Processor -- start 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.056+0000 7fa999889640 1 -- start start 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.056+0000 7fa999889640 1 librados: setting wanted keys 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.056+0000 7fa999889640 1 librados: calling monclient init 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.056+0000 7fa999889640 10 monclient: init 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.056+0000 7fa999889640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:04.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.056+0000 7fa999889640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:04.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.057+0000 7fa999889640 10 monclient(hunting): picked mon.noname-a con 0x7fa98c006940 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:04.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.059+0000 7fa999889640 10 monclient(hunting): picked mon.noname-b con 0x7fa98c02b090 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:04.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.059+0000 7fa999889640 10 monclient(hunting): picked mon.noname-c con 0x7fa98c0eec90 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:04.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.060+0000 7fa999889640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98c02f940 con 0x7fa98c006940 2024-03-29T11:40:04.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.060+0000 7fa999889640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98c02ea70 con 0x7fa98c02b090 2024-03-29T11:40:04.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.060+0000 7fa999889640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98c030960 con 0x7fa98c0eec90 2024-03-29T11:40:04.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.060+0000 7fa999889640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:04.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.060+0000 7fa999889640 10 monclient(hunting): authenticate will time out at 1183.509191s 2024-03-29T11:40:04.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.063+0000 7fa998887640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa98c006940 0x7fa98c030d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42153/0 (socket says 172.21.15.37:42153) 2024-03-29T11:40:04.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.063+0000 7fa998887640 1 -- 172.21.15.37:0/2620454408 learned_addr learned my addr 172.21.15.37:0/2620454408 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:04.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.063+0000 7fa9917fa640 10 client.?.objecter ms_handle_connect 0x7fa98c0eec90 2024-03-29T11:40:04.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.064+0000 7fa9917fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:04.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.066+0000 7fa9917fa640 10 client.?.objecter ms_handle_connect 0x7fa98c006940 2024-03-29T11:40:04.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.066+0000 7fa9917fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:04.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.066+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486181066 0 0) 0x7fa98c030960 con 0x7fa98c0eec90 2024-03-29T11:40:04.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.066+0000 7fa9917fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.066+0000 7fa9917fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.066+0000 7fa9917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.066+0000 7fa9917fa640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:40:04.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.066+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa978004eb0 con 0x7fa98c0eec90 2024-03-29T11:40:04.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.066+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039875589 0 0) 0x7fa98c02f940 con 0x7fa98c006940 2024-03-29T11:40:04.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.067+0000 7fa9917fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.067+0000 7fa9917fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.067+0000 7fa9917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.067+0000 7fa9917fa640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.067+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa98c030960 con 0x7fa98c006940 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.067+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2224667240 0 0) 0x7fa98c030960 con 0x7fa98c006940 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.067+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa98c02f940 con 0x7fa98c006940 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.067+0000 7fa9917fa640 10 client.?.objecter ms_handle_connect 0x7fa98c02b090 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.067+0000 7fa9917fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.067+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa988001bb0 con 0x7fa98c006940 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 10 monclient(hunting): dump: 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:04.074 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526833098 0 0) 0x7fa98c02ea70 con 0x7fa98c02b090 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa98c030960 con 0x7fa98c02b090 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 816675396 0 0) 0x7fa978004eb0 con 0x7fa98c0eec90 2024-03-29T11:40:04.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa98c02ea70 con 0x7fa98c0eec90 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa99404fdd0 con 0x7fa98c0eec90 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.068+0000 7fa9917fa640 10 monclient(hunting): dump: 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:04.076 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1649362444 0 0) 0x7fa98c030960 con 0x7fa98c02b090 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa978004eb0 con 0x7fa98c02b090 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa980001750 con 0x7fa98c02b090 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 10 monclient(hunting): dump: 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:04.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2208544575 0 0) 0x7fa98c02ea70 con 0x7fa98c0eec90 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 >> v1:172.21.15.146:6789/0 conn(0x7fa98c02b090 legacy=0x7fa98c02e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 >> v1:172.21.15.37:6789/0 conn(0x7fa98c006940 legacy=0x7fa98c030d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 10 monclient: _finish_hunting 0 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 1 monclient: found mon.c 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c0df220 con 0x7fa98c0eec90 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 10 monclient: _finish_auth 0 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 10 monclient: _check_auth_tickets 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa999889640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa999889640 10 monclient: _renew_subs 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa999889640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa999889640 1 -- 172.21.15.37:0/2620454408 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98c0df470 con 0x7fa98c0eec90 2024-03-29T11:40:04.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa999889640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa999889640 10 monclient: _renew_subs 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa999889640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa999889640 1 -- 172.21.15.37:0/2620454408 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98c18c190 con 0x7fa98c0eec90 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa999889640 1 librados: init done 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa99404f400 con 0x7fa98c0eec90 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fa994075710 con 0x7fa98c0eec90 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.069+0000 7fa9917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.070+0000 7fa9917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.070+0000 7fa9917fa640 10 monclient: dump: 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.079 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:04.080 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:04.080 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:04.080 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:04.080 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:04.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.070+0000 7fa992ffd640 4 set_mon_vals no callback set 2024-03-29T11:40:04.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.071+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fa994067bf0 con 0x7fa98c0eec90 2024-03-29T11:40:04.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.072+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fa994075870 con 0x7fa98c0eec90 2024-03-29T11:40:04.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.072+0000 7fa9917fa640 10 client.4436.objecter ms_dispatch 0x7fa994000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:04.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.072+0000 7fa9917fa640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:04.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.072+0000 7fa9917fa640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:04.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.072+0000 7fa9917fa640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-29T11:40:04.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.073+0000 7fa998887640 10 monclient: get_auth_request con 0x7fa97806dfc0 auth_method 0 2024-03-29T11:40:04.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.073+0000 7fa999889640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:04.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.073+0000 7fa9917fa640 10 client.4436.objecter ms_handle_connect 0x7fa97806dfc0 2024-03-29T11:40:04.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.073+0000 7fa999889640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:04.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.073+0000 7fa999889640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:04.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.073+0000 7fa999889640 1 -- 172.21.15.37:0/2620454408 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa974005080 con 0x7fa98c0eec90 2024-03-29T11:40:04.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.082+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9940b9010 con 0x7fa98c0eec90 2024-03-29T11:40:04.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.082+0000 7fa9917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:04.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.082+0000 7fa9917fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:04.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.419+0000 7fa999889640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:04.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.419+0000 7fa999889640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:04.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.419+0000 7fa999889640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:04.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.419+0000 7fa999889640 1 -- 172.21.15.37:0/2620454408 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa974005330 con 0x7fa98c0eec90 2024-03-29T11:40:04.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.420+0000 7fa9917fa640 1 -- 172.21.15.37:0/2620454408 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fa9940b9010 con 0x7fa98c0eec90 2024-03-29T11:40:04.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.420+0000 7fa9917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:04.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.420+0000 7fa9917fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:04.422 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:40:04.422 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:04.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.424+0000 7fa999889640 10 librados: watch_flush enter 2024-03-29T11:40:04.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.424+0000 7fa999889640 10 librados: watch_flush exit 2024-03-29T11:40:04.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.424+0000 7fa999889640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-29T11:40:04.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.424+0000 7fa999889640 10 client.4436.objecter successfully canceled tick 2024-03-29T11:40:04.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.425+0000 7fa999889640 1 -- 172.21.15.37:0/2620454408 >> v1:172.21.15.146:6816/2899149139 conn(0x7fa97806dfc0 legacy=0x7fa978070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.425+0000 7fa999889640 10 monclient: shutdown 2024-03-29T11:40:04.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.425+0000 7fa999889640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:04.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.425+0000 7fa999889640 1 -- 172.21.15.37:0/2620454408 >> v1:172.21.15.37:6790/0 conn(0x7fa98c0eec90 legacy=0x7fa98c02f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.426+0000 7fa999889640 1 -- 172.21.15.37:0/2620454408 shutdown_connections 2024-03-29T11:40:04.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.426+0000 7fa999889640 1 -- 172.21.15.37:0/2620454408 >> 172.21.15.37:0/2620454408 conn(0x7fa98c0058f0 msgr2=0x7fa98c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:04.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.426+0000 7fa999889640 1 -- 172.21.15.37:0/2620454408 shutdown_connections 2024-03-29T11:40:04.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.426+0000 7fa999889640 1 -- 172.21.15.37:0/2620454408 wait complete. 2024-03-29T11:40:04.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:04.426+0000 7fa999889640 1 librados: shutdown 2024-03-29T11:40:04.446 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:11.252 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:11.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.399+0000 7fb4040c1640 10 monclient: get_monmap_and_config 2024-03-29T11:40:11.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.399+0000 7fb4040c1640 10 monclient: build_initial_monmap 2024-03-29T11:40:11.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.399+0000 7fb4040c1640 10 monclient: monmap: 2024-03-29T11:40:11.406 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:11.406 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:11.406 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:40:11.401238+0000 2024-03-29T11:40:11.406 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:40:11.401238+0000 2024-03-29T11:40:11.406 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:11.406 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:11.406 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:11.406 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:40:11.406 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:40:11.406 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:11.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.402+0000 7fb4040c1640 1 Processor -- start 2024-03-29T11:40:11.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.403+0000 7fb4040c1640 1 -- start start 2024-03-29T11:40:11.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.403+0000 7fb4040c1640 10 monclient: init 2024-03-29T11:40:11.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.404+0000 7fb4040c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:11.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.404+0000 7fb4040c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:11.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.404+0000 7fb4040c1640 10 monclient(hunting): picked mon.noname-a con 0x7fb3fc0a9750 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:11.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.404+0000 7fb4040c1640 10 monclient(hunting): picked mon.noname-c con 0x7fb3fc149ac0 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:11.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.404+0000 7fb4040c1640 10 monclient(hunting): picked mon.noname-b con 0x7fb3fc14fe50 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:11.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.404+0000 7fb4040c1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc0aa4a0 con 0x7fb3fc0a9750 2024-03-29T11:40:11.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.404+0000 7fb4040c1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc14a680 con 0x7fb3fc149ac0 2024-03-29T11:40:11.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.404+0000 7fb4040c1640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc0af290 con 0x7fb3fc14fe50 2024-03-29T11:40:11.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.404+0000 7fb4040c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:11.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.404+0000 7fb4040c1640 10 monclient(hunting): authenticate will time out at 1190.854136s 2024-03-29T11:40:11.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.407+0000 7fb40265e640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fb3fc14fe50 0x7fb3fc152240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:49041/0 (socket says 172.21.15.37:49041) 2024-03-29T11:40:11.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.407+0000 7fb40265e640 1 -- 172.21.15.37:0/909679850 learned_addr learned my addr 172.21.15.37:0/909679850 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:11.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.408+0000 7fb400e5b640 1 -- 172.21.15.37:0/909679850 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212198551 0 0) 0x7fb3fc0af290 con 0x7fb3fc14fe50 2024-03-29T11:40:11.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.408+0000 7fb400e5b640 10 monclient(hunting): _negotiate 2024-03-29T11:40:11.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.408+0000 7fb400e5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:11.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.408+0000 7fb400e5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:11.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.408+0000 7fb400e5b640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:40:11.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.408+0000 7fb400e5b640 1 -- 172.21.15.37:0/909679850 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e8003520 con 0x7fb3fc14fe50 2024-03-29T11:40:11.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.409+0000 7fb400e5b640 1 -- 172.21.15.37:0/909679850 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4225541713 0 0) 0x7fb3e8003520 con 0x7fb3fc14fe50 2024-03-29T11:40:11.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.409+0000 7fb400e5b640 1 -- 172.21.15.37:0/909679850 >> v1:172.21.15.146:6789/0 conn(0x7fb3fc149ac0 legacy=0x7fb3fc149f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:11.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 1 -- 172.21.15.37:0/909679850 >> v1:172.21.15.37:6789/0 conn(0x7fb3fc0a9750 legacy=0x7fb3fc1436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 10 monclient: _finish_hunting 0 2024-03-29T11:40:11.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 1 monclient: found mon.noname-b 2024-03-29T11:40:11.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:11.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:40:11.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 1 -- 172.21.15.37:0/909679850 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc0afb40 con 0x7fb3fc14fe50 2024-03-29T11:40:11.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 10 monclient: _finish_auth 0 2024-03-29T11:40:11.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 10 monclient: _check_auth_tickets 2024-03-29T11:40:11.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:11.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 1 -- 172.21.15.37:0/909679850 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fb3f4001a10 con 0x7fb3fc14fe50 2024-03-29T11:40:11.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:11.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:11.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 10 monclient: dump: 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb4040c1640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb4040c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 1 -- 172.21.15.37:0/909679850 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3f4001be0 con 0x7fb3fc14fe50 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:11.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 1 -- 172.21.15.37:0/909679850 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fb3f4002030 con 0x7fb3fc14fe50 2024-03-29T11:40:11.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb4040c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:11.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb4040c1640 4 set_mon_vals no callback set 2024-03-29T11:40:11.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:11.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:11.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.410+0000 7fb400e5b640 10 monclient: dump: 2024-03-29T11:40:11.416 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:11.416 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:11.416 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:11.416 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:11.416 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:11.416 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:11.416 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:11.416 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:11.416 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:11.417 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:11.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.413+0000 7fb4040c1640 10 monclient: shutdown 2024-03-29T11:40:11.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.413+0000 7fb4040c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:11.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.413+0000 7fb4040c1640 1 -- 172.21.15.37:0/909679850 >> v1:172.21.15.37:6790/0 conn(0x7fb3fc14fe50 legacy=0x7fb3fc152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.414+0000 7fb4040c1640 1 -- 172.21.15.37:0/909679850 shutdown_connections 2024-03-29T11:40:11.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.414+0000 7fb4040c1640 1 -- 172.21.15.37:0/909679850 >> 172.21.15.37:0/909679850 conn(0x7fb3fc0b35e0 msgr2=0x7fb3fc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:11.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.417+0000 7fb4040c1640 1 -- 172.21.15.37:0/909679850 shutdown_connections 2024-03-29T11:40:11.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.417+0000 7fb4040c1640 1 -- 172.21.15.37:0/909679850 wait complete. 2024-03-29T11:40:11.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.418+0000 7fb4040c1640 10 monclient: build_initial_monmap 2024-03-29T11:40:11.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.419+0000 7fb4040c1640 10 monclient: monmap: 2024-03-29T11:40:11.422 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:11.422 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:11.422 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:40:11.422 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:40:11.422 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:11.422 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:11.422 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:11.422 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:40:11.423 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:40:11.423 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:11.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.419+0000 7fb4040c1640 1 librados: starting msgr at 2024-03-29T11:40:11.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.419+0000 7fb4040c1640 1 librados: starting objecter 2024-03-29T11:40:11.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.419+0000 7fb4040c1640 1 Processor -- start 2024-03-29T11:40:11.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.420+0000 7fb4040c1640 1 -- start start 2024-03-29T11:40:11.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.420+0000 7fb4040c1640 1 librados: setting wanted keys 2024-03-29T11:40:11.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.420+0000 7fb4040c1640 1 librados: calling monclient init 2024-03-29T11:40:11.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.420+0000 7fb4040c1640 10 monclient: init 2024-03-29T11:40:11.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.421+0000 7fb4040c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:11.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.424+0000 7fb4040c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:11.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.425+0000 7fb4040c1640 10 monclient(hunting): picked mon.noname-a con 0x7fb3fc0a9750 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:11.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.425+0000 7fb4040c1640 10 monclient(hunting): picked mon.noname-b con 0x7fb3fc149ac0 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:11.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.425+0000 7fb4040c1640 10 monclient(hunting): picked mon.noname-c con 0x7fb3fc14fe50 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:11.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.425+0000 7fb4040c1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc13d4c0 con 0x7fb3fc0a9750 2024-03-29T11:40:11.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.425+0000 7fb4040c1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc13c220 con 0x7fb3fc149ac0 2024-03-29T11:40:11.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.425+0000 7fb4040c1640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc13d0a0 con 0x7fb3fc14fe50 2024-03-29T11:40:11.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.425+0000 7fb4040c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:11.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.425+0000 7fb4040c1640 10 monclient(hunting): authenticate will time out at 1190.875290s 2024-03-29T11:40:11.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.428+0000 7fb401e5d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fb3fc0a9750 0x7fb3fc13c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55133/0 (socket says 172.21.15.37:55133) 2024-03-29T11:40:11.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.428+0000 7fb401e5d640 1 -- 172.21.15.37:0/758344337 learned_addr learned my addr 172.21.15.37:0/758344337 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:11.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.428+0000 7fb3f27fc640 10 client.?.objecter ms_handle_connect 0x7fb3fc0a9750 2024-03-29T11:40:11.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.429+0000 7fb3f27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:11.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.429+0000 7fb3f27fc640 10 client.?.objecter ms_handle_connect 0x7fb3fc14fe50 2024-03-29T11:40:11.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.429+0000 7fb3f27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:11.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.429+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004741560 0 0) 0x7fb3fc13d4c0 con 0x7fb3fc0a9750 2024-03-29T11:40:11.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.429+0000 7fb3f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:11.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.429+0000 7fb3f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:11.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.429+0000 7fb3f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:11.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.429+0000 7fb3f27fc640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:40:11.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.429+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e0004eb0 con 0x7fb3fc0a9750 2024-03-29T11:40:11.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.429+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055260752 0 0) 0x7fb3fc13d0a0 con 0x7fb3fc14fe50 2024-03-29T11:40:11.435 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.430+0000 7fb3f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:11.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.430+0000 7fb3f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:11.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.430+0000 7fb3f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:11.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.430+0000 7fb3f27fc640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:40:11.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.430+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3fc13d4c0 con 0x7fb3fc14fe50 2024-03-29T11:40:11.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.430+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1873036873 0 0) 0x7fb3e0004eb0 con 0x7fb3fc0a9750 2024-03-29T11:40:11.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.430+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3fc13d0a0 con 0x7fb3fc0a9750 2024-03-29T11:40:11.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fb3f8001970 con 0x7fb3fc0a9750 2024-03-29T11:40:11.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:11.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:11.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 10 monclient(hunting): dump: 2024-03-29T11:40:11.436 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:11.436 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:11.436 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:11.436 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4066625153 0 0) 0x7fb3fc13d0a0 con 0x7fb3fc0a9750 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 >> v1:172.21.15.37:6790/0 conn(0x7fb3fc14fe50 legacy=0x7fb3fc1db550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 >> v1:172.21.15.146:6789/0 conn(0x7fb3fc149ac0 legacy=0x7fb3fc13a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 1 monclient: found mon.a 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:11.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc13aa60 con 0x7fb3fc0a9750 2024-03-29T11:40:11.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:11.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:11.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.431+0000 7fb3f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:11.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.432+0000 7fb4040c1640 5 monclient: authenticate success, global_id 4404 2024-03-29T11:40:11.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.432+0000 7fb4040c1640 10 monclient: _renew_subs 2024-03-29T11:40:11.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.432+0000 7fb4040c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:11.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.432+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3f8001ad0 con 0x7fb3fc0a9750 2024-03-29T11:40:11.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.433+0000 7fb4040c1640 1 -- 172.21.15.37:0/758344337 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc13acb0 con 0x7fb3fc0a9750 2024-03-29T11:40:11.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.433+0000 7fb4040c1640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:11.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.433+0000 7fb3f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:11.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.433+0000 7fb3f3fff640 4 set_mon_vals no callback set 2024-03-29T11:40:11.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.434+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fb3f80038c0 con 0x7fb3fc0a9750 2024-03-29T11:40:11.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.434+0000 7fb3f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:11.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.434+0000 7fb3f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:11.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.434+0000 7fb3f27fc640 10 monclient: dump: 2024-03-29T11:40:11.440 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:11.440 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:11.440 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:11.440 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:11.440 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:11.440 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:11.440 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:11.440 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:11.440 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:11.440 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:11.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.434+0000 7fb4040c1640 10 monclient: _renew_subs 2024-03-29T11:40:11.441 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.434+0000 7fb4040c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:11.441 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.434+0000 7fb4040c1640 1 -- 172.21.15.37:0/758344337 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc0ac6e0 con 0x7fb3fc0a9750 2024-03-29T11:40:11.441 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.434+0000 7fb4040c1640 1 librados: init done 2024-03-29T11:40:11.441 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.436+0000 7fb3dffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:11.441 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.436+0000 7fb3dffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:11.441 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.436+0000 7fb3dffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:11.441 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.436+0000 7fb3dffff640 1 -- 172.21.15.37:0/758344337 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc13d230 con 0x7fb3fc0a9750 2024-03-29T11:40:11.441 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.439+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fb3f8003a20 con 0x7fb3fc0a9750 2024-03-29T11:40:11.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.441+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fb3f8085ad0 con 0x7fb3fc0a9750 2024-03-29T11:40:11.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.441+0000 7fb3f27fc640 10 client.4404.objecter ms_dispatch 0x7fb3fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:11.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.441+0000 7fb3f27fc640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:11.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.441+0000 7fb40165c640 10 monclient: get_auth_request con 0x7fb3e006dd20 auth_method 0 2024-03-29T11:40:11.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.441+0000 7fb3f27fc640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:11.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.441+0000 7fb3f27fc640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-29T11:40:11.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.441+0000 7fb3f27fc640 10 client.4404.objecter ms_handle_connect 0x7fb3e006dd20 2024-03-29T11:40:11.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.447+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb3f8050380 con 0x7fb3fc0a9750 2024-03-29T11:40:11.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.447+0000 7fb3f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:11.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.447+0000 7fb3f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:11.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.782+0000 7fb3dffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:11.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.782+0000 7fb3dffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:11.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.782+0000 7fb3dffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:11.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.782+0000 7fb3dffff640 1 -- 172.21.15.37:0/758344337 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3fc075f10 con 0x7fb3fc0a9750 2024-03-29T11:40:11.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.782+0000 7fb3f27fc640 1 -- 172.21.15.37:0/758344337 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fb3f80522b0 con 0x7fb3fc0a9750 2024-03-29T11:40:11.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.782+0000 7fb3f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:11.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.782+0000 7fb3f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:11.785 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:40:11.785 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:11.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.786+0000 7fb3dffff640 10 librados: watch_flush enter 2024-03-29T11:40:11.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.788+0000 7fb3dffff640 10 librados: watch_flush exit 2024-03-29T11:40:11.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.788+0000 7fb3dffff640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-29T11:40:11.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.788+0000 7fb3dffff640 10 client.4404.objecter successfully canceled tick 2024-03-29T11:40:11.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.788+0000 7fb3dffff640 1 -- 172.21.15.37:0/758344337 >> v1:172.21.15.146:6816/2899149139 conn(0x7fb3e006dd20 legacy=0x7fb3e00701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.788+0000 7fb3dffff640 10 monclient: shutdown 2024-03-29T11:40:11.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.788+0000 7fb3dffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:11.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.788+0000 7fb3dffff640 1 -- 172.21.15.37:0/758344337 >> v1:172.21.15.37:6789/0 conn(0x7fb3fc0a9750 legacy=0x7fb3fc13c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.790+0000 7fb3dffff640 1 -- 172.21.15.37:0/758344337 shutdown_connections 2024-03-29T11:40:11.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.790+0000 7fb3dffff640 1 -- 172.21.15.37:0/758344337 >> 172.21.15.37:0/758344337 conn(0x7fb3fc0b35e0 msgr2=0x7fb3fc0ae8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:11.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.791+0000 7fb3dffff640 1 -- 172.21.15.37:0/758344337 shutdown_connections 2024-03-29T11:40:11.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.791+0000 7fb3dffff640 1 -- 172.21.15.37:0/758344337 wait complete. 2024-03-29T11:40:11.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:11.792+0000 7fb3dffff640 1 librados: shutdown 2024-03-29T11:40:11.812 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:18.619 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:18.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.769+0000 7f3d4c74d640 10 monclient: get_monmap_and_config 2024-03-29T11:40:18.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.769+0000 7f3d4c74d640 10 monclient: build_initial_monmap 2024-03-29T11:40:18.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.769+0000 7f3d4c74d640 10 monclient: monmap: 2024-03-29T11:40:18.772 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:18.772 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:18.772 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:40:18.771431+0000 2024-03-29T11:40:18.772 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:40:18.771431+0000 2024-03-29T11:40:18.772 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:18.772 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:18.773 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:18.773 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:40:18.773 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:40:18.773 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:18.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.770+0000 7f3d4c74d640 1 Processor -- start 2024-03-29T11:40:18.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.770+0000 7f3d4c74d640 1 -- start start 2024-03-29T11:40:18.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.770+0000 7f3d4c74d640 10 monclient: init 2024-03-29T11:40:18.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.772+0000 7f3d4c74d640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:18.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.772+0000 7f3d4c74d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:18.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.772+0000 7f3d4c74d640 10 monclient(hunting): picked mon.noname-c con 0x7f3d441457f0 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:18.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.772+0000 7f3d4c74d640 10 monclient(hunting): picked mon.noname-a con 0x7f3d4413f650 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:18.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.772+0000 7f3d4c74d640 10 monclient(hunting): picked mon.noname-b con 0x7f3d440a9d70 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:18.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.772+0000 7f3d4c74d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d440aaef0 con 0x7f3d4413f650 2024-03-29T11:40:18.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.772+0000 7f3d4c74d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d44145bd0 con 0x7f3d441457f0 2024-03-29T11:40:18.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.772+0000 7f3d4c74d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d44145eb0 con 0x7f3d440a9d70 2024-03-29T11:40:18.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.772+0000 7f3d4c74d640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:18.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.772+0000 7f3d4c74d640 10 monclient(hunting): authenticate will time out at 1198.222186s 2024-03-29T11:40:18.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.772+0000 7f3d4acea640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f3d440a9d70 0x7f3d4414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:35532/0 (socket says 172.21.15.37:35532) 2024-03-29T11:40:18.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.772+0000 7f3d4acea640 1 -- 172.21.15.37:0/1760348600 learned_addr learned my addr 172.21.15.37:0/1760348600 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:18.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.772+0000 7f3d494e7640 1 -- 172.21.15.37:0/1760348600 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991206574 0 0) 0x7f3d44145eb0 con 0x7f3d440a9d70 2024-03-29T11:40:18.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient(hunting): _negotiate 2024-03-29T11:40:18.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:18.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:18.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 1 -- 172.21.15.37:0/1760348600 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d30003520 con 0x7f3d440a9d70 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 1 -- 172.21.15.37:0/1760348600 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763706737 0 0) 0x7f3d44145bd0 con 0x7f3d441457f0 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient(hunting): _negotiate 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 1 -- 172.21.15.37:0/1760348600 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d44145eb0 con 0x7f3d441457f0 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 1 -- 172.21.15.37:0/1760348600 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1949041350 0 0) 0x7f3d44145eb0 con 0x7f3d441457f0 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 1 -- 172.21.15.37:0/1760348600 >> v1:172.21.15.37:6790/0 conn(0x7f3d440a9d70 legacy=0x7f3d4414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 1 -- 172.21.15.37:0/1760348600 >> v1:172.21.15.37:6789/0 conn(0x7f3d4413f650 legacy=0x7f3d440b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient: _finish_hunting 0 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 1 monclient: found mon.noname-c 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:40:18.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 1 -- 172.21.15.37:0/1760348600 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d44146340 con 0x7f3d441457f0 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient: _finish_auth 0 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient: _check_auth_tickets 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 1 -- 172.21.15.37:0/1760348600 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3d340015a0 con 0x7f3d441457f0 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d4c74d640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d4c74d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d494e7640 10 monclient: dump: 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:18.778 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:18.779 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:18.779 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:18.779 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:18.779 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:18.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.773+0000 7f3d4c74d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:18.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.774+0000 7f3d494e7640 1 -- 172.21.15.37:0/1760348600 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d34001700 con 0x7f3d441457f0 2024-03-29T11:40:18.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.774+0000 7f3d494e7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:18.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.774+0000 7f3d494e7640 1 -- 172.21.15.37:0/1760348600 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3d34001b50 con 0x7f3d441457f0 2024-03-29T11:40:18.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.774+0000 7f3d494e7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:18.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.774+0000 7f3d494e7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:18.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.774+0000 7f3d494e7640 10 monclient: dump: 2024-03-29T11:40:18.779 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:18.779 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:18.779 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:18.780 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:18.780 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:18.780 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:18.780 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:18.780 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:18.780 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:18.780 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:18.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.774+0000 7f3d4c74d640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:18.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.774+0000 7f3d4c74d640 4 set_mon_vals no callback set 2024-03-29T11:40:18.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.775+0000 7f3d4c74d640 10 monclient: shutdown 2024-03-29T11:40:18.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.775+0000 7f3d4c74d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:18.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.775+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1760348600 >> v1:172.21.15.146:6789/0 conn(0x7f3d441457f0 legacy=0x7f3d4413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:18.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.775+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1760348600 shutdown_connections 2024-03-29T11:40:18.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.775+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1760348600 >> 172.21.15.37:0/1760348600 conn(0x7f3d44066a00 msgr2=0x7f3d44066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:18.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.775+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1760348600 shutdown_connections 2024-03-29T11:40:18.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.775+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1760348600 wait complete. 2024-03-29T11:40:18.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.776+0000 7f3d4c74d640 10 monclient: build_initial_monmap 2024-03-29T11:40:18.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.776+0000 7f3d4c74d640 10 monclient: monmap: 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.776+0000 7f3d4c74d640 1 librados: starting msgr at 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.776+0000 7f3d4c74d640 1 librados: starting objecter 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.777+0000 7f3d4c74d640 1 Processor -- start 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.777+0000 7f3d4c74d640 1 -- start start 2024-03-29T11:40:18.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.777+0000 7f3d4c74d640 1 librados: setting wanted keys 2024-03-29T11:40:18.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.777+0000 7f3d4c74d640 1 librados: calling monclient init 2024-03-29T11:40:18.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.777+0000 7f3d4c74d640 10 monclient: init 2024-03-29T11:40:18.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.777+0000 7f3d4c74d640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:18.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.777+0000 7f3d4c74d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:18.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.777+0000 7f3d4c74d640 10 monclient(hunting): picked mon.noname-b con 0x7f3d440a9d70 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:18.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.777+0000 7f3d4c74d640 10 monclient(hunting): picked mon.noname-c con 0x7f3d4413f650 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:18.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d4c74d640 10 monclient(hunting): picked mon.noname-a con 0x7f3d441457f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:18.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d4c74d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d440af620 con 0x7f3d441457f0 2024-03-29T11:40:18.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d4c74d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d440af9d0 con 0x7f3d440a9d70 2024-03-29T11:40:18.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d4acea640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f3d441457f0 0x7f3d441e2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:38172/0 (socket says 172.21.15.37:38172) 2024-03-29T11:40:18.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d4acea640 1 -- 172.21.15.37:0/1170257762 learned_addr learned my addr 172.21.15.37:0/1170257762 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:18.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1170257762 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d440afb60 con 0x7f3d4413f650 2024-03-29T11:40:18.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d4c74d640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:18.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d4c74d640 10 monclient(hunting): authenticate will time out at 1198.228322s 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d3affd640 10 client.?.objecter ms_handle_connect 0x7f3d441457f0 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d3affd640 10 client.?.objecter ms_handle_connect 0x7f3d4413f650 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808925178 0 0) 0x7f3d440afb60 con 0x7f3d4413f650 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d3affd640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.778+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d28004eb0 con 0x7f3d4413f650 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.779+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1418750060 0 0) 0x7f3d28004eb0 con 0x7f3d4413f650 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.779+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d440afb60 con 0x7f3d4413f650 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.779+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3d40001930 con 0x7f3d4413f650 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.779+0000 7f3d3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:18.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.779+0000 7f3d3affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.779+0000 7f3d3affd640 10 monclient(hunting): dump: 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.779+0000 7f3d3affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078087602 0 0) 0x7f3d440af620 con 0x7f3d441457f0 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:18.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d28004eb0 con 0x7f3d441457f0 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3581117051 0 0) 0x7f3d440afb60 con 0x7f3d4413f650 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 >> v1:172.21.15.146:6789/0 conn(0x7f3d440a9d70 legacy=0x7f3d440acf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 >> v1:172.21.15.37:6789/0 conn(0x7f3d441457f0 legacy=0x7f3d441e2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 1 monclient: found mon.c 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d440ad660 con 0x7f3d4413f650 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 10 monclient: _finish_auth 0 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:18.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d4c74d640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:40:18.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d4c74d640 10 monclient: _renew_subs 2024-03-29T11:40:18.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d4c74d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:18.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1170257762 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d440ad8b0 con 0x7f3d4413f650 2024-03-29T11:40:18.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d4c74d640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:18.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d4c74d640 10 monclient: _renew_subs 2024-03-29T11:40:18.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d4c74d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:18.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1170257762 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d440b0d80 con 0x7f3d4413f650 2024-03-29T11:40:18.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.780+0000 7f3d4c74d640 1 librados: init done 2024-03-29T11:40:18.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.781+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d40001b00 con 0x7f3d4413f650 2024-03-29T11:40:18.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.781+0000 7f3d3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:18.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.781+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3d40003a40 con 0x7f3d4413f650 2024-03-29T11:40:18.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.781+0000 7f3d3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:18.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.781+0000 7f3d3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:18.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.781+0000 7f3d48ce6640 4 set_mon_vals no callback set 2024-03-29T11:40:18.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.781+0000 7f3d3affd640 10 monclient: dump: 2024-03-29T11:40:18.794 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:18.794 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:18.794 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:18.794 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:18.794 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:18.794 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:18.794 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:18.794 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:18.794 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:18.794 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:18.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.782+0000 7f3d4c74d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:18.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.782+0000 7f3d4c74d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:18.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.782+0000 7f3d4c74d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:18.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.782+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1170257762 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d2c005080 con 0x7f3d4413f650 2024-03-29T11:40:18.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.785+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f3d40003ba0 con 0x7f3d4413f650 2024-03-29T11:40:18.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.785+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f3d400838c0 con 0x7f3d4413f650 2024-03-29T11:40:18.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.785+0000 7f3d3affd640 10 client.4448.objecter ms_dispatch 0x7f3d44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:18.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.785+0000 7f3d3affd640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:18.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.785+0000 7f3d3affd640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:18.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.785+0000 7f3d3affd640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T11:40:18.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.785+0000 7f3d4a4e9640 10 monclient: get_auth_request con 0x7f3d28076570 auth_method 0 2024-03-29T11:40:18.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.786+0000 7f3d3affd640 10 client.4448.objecter ms_handle_connect 0x7f3d28076570 2024-03-29T11:40:18.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.792+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d4004efd0 con 0x7f3d4413f650 2024-03-29T11:40:18.797 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.792+0000 7f3d3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:18.797 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:18.792+0000 7f3d3affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:19.183 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.180+0000 7f3d4c74d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:19.183 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.180+0000 7f3d4c74d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:19.183 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.180+0000 7f3d4c74d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:19.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.180+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1170257762 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d2c005330 con 0x7f3d4413f650 2024-03-29T11:40:19.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.182+0000 7f3d3affd640 1 -- 172.21.15.37:0/1170257762 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f3d400510d0 con 0x7f3d4413f650 2024-03-29T11:40:19.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.182+0000 7f3d3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:19.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.182+0000 7f3d3affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:19.185 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:40:19.185 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:19.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.185+0000 7f3d4c74d640 10 librados: watch_flush enter 2024-03-29T11:40:19.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.185+0000 7f3d4c74d640 10 librados: watch_flush exit 2024-03-29T11:40:19.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.185+0000 7f3d4c74d640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T11:40:19.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.185+0000 7f3d4c74d640 10 client.4448.objecter successfully canceled tick 2024-03-29T11:40:19.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.185+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1170257762 >> v1:172.21.15.146:6816/2899149139 conn(0x7f3d28076570 legacy=0x7f3d28078a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.185+0000 7f3d4c74d640 10 monclient: shutdown 2024-03-29T11:40:19.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.185+0000 7f3d4c74d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:19.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.185+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1170257762 >> v1:172.21.15.37:6790/0 conn(0x7f3d4413f650 legacy=0x7f3d44139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.185+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1170257762 shutdown_connections 2024-03-29T11:40:19.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.185+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1170257762 >> 172.21.15.37:0/1170257762 conn(0x7f3d44066a00 msgr2=0x7f3d44066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:19.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.185+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1170257762 shutdown_connections 2024-03-29T11:40:19.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.185+0000 7f3d4c74d640 1 -- 172.21.15.37:0/1170257762 wait complete. 2024-03-29T11:40:19.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:19.186+0000 7f3d4c74d640 1 librados: shutdown 2024-03-29T11:40:19.203 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:26.010 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:26.160 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.158+0000 7efc7a52b640 10 monclient: get_monmap_and_config 2024-03-29T11:40:26.160 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.158+0000 7efc7a52b640 10 monclient: build_initial_monmap 2024-03-29T11:40:26.160 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.158+0000 7efc7a52b640 10 monclient: monmap: 2024-03-29T11:40:26.160 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:26.160 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:26.160 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:40:26.159264+0000 2024-03-29T11:40:26.160 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:40:26.159264+0000 2024-03-29T11:40:26.160 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:26.161 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:26.161 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:26.161 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:40:26.161 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:40:26.161 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:26.164 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.162+0000 7efc7a52b640 1 Processor -- start 2024-03-29T11:40:26.164 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.162+0000 7efc7a52b640 1 -- start start 2024-03-29T11:40:26.164 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.162+0000 7efc7a52b640 10 monclient: init 2024-03-29T11:40:26.164 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.163+0000 7efc7a52b640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:26.164 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.163+0000 7efc7a52b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:26.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.163+0000 7efc7a52b640 10 monclient(hunting): picked mon.noname-b con 0x7efc740b1f60 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:26.168 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.165+0000 7efc7a52b640 10 monclient(hunting): picked mon.noname-a con 0x7efc74071eb0 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:26.168 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.166+0000 7efc7a52b640 10 monclient(hunting): picked mon.noname-c con 0x7efc74143d10 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:26.168 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.166+0000 7efc7a52b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7406ef80 con 0x7efc74071eb0 2024-03-29T11:40:26.168 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.166+0000 7efc7a52b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7406f260 con 0x7efc74143d10 2024-03-29T11:40:26.168 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.166+0000 7efc7a52b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc740b40c0 con 0x7efc740b1f60 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.166+0000 7efc7a52b640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.166+0000 7efc7a52b640 10 monclient(hunting): authenticate will time out at 1205.615539s 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.166+0000 7efc737fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7efc74071eb0 0x7efc74146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41299/0 (socket says 172.21.15.37:41299) 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.166+0000 7efc737fe640 1 -- 172.21.15.37:0/1818774703 learned_addr learned my addr 172.21.15.37:0/1818774703 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.167+0000 7efc72ffd640 1 -- 172.21.15.37:0/1818774703 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941978534 0 0) 0x7efc740b40c0 con 0x7efc740b1f60 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.167+0000 7efc72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.167+0000 7efc72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.167+0000 7efc72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.167+0000 7efc72ffd640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.167+0000 7efc72ffd640 1 -- 172.21.15.37:0/1818774703 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc5c003520 con 0x7efc740b1f60 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.167+0000 7efc72ffd640 1 -- 172.21.15.37:0/1818774703 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577702967 0 0) 0x7efc7406ef80 con 0x7efc74071eb0 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.167+0000 7efc72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.167+0000 7efc72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.167+0000 7efc72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.167+0000 7efc72ffd640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.167+0000 7efc72ffd640 1 -- 172.21.15.37:0/1818774703 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc740b40c0 con 0x7efc74071eb0 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.168+0000 7efc72ffd640 1 -- 172.21.15.37:0/1818774703 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2375615321 0 0) 0x7efc5c003520 con 0x7efc740b1f60 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.168+0000 7efc72ffd640 1 -- 172.21.15.37:0/1818774703 >> v1:172.21.15.146:6789/0 conn(0x7efc74143d10 legacy=0x7efc74146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.168+0000 7efc72ffd640 1 -- 172.21.15.37:0/1818774703 >> v1:172.21.15.37:6789/0 conn(0x7efc74071eb0 legacy=0x7efc74146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.168+0000 7efc72ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.168+0000 7efc72ffd640 1 monclient: found mon.noname-b 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.168+0000 7efc72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc72ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc72ffd640 1 -- 172.21.15.37:0/1818774703 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc74155d70 con 0x7efc740b1f60 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc72ffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc72ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc72ffd640 1 -- 172.21.15.37:0/1818774703 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7efc64001690 con 0x7efc740b1f60 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc7a52b640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc72ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc72ffd640 10 monclient: dump: 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc7a52b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc72ffd640 1 -- 172.21.15.37:0/1818774703 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc64001870 con 0x7efc740b1f60 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc7a52b640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.169+0000 7efc7a52b640 4 set_mon_vals no callback set 2024-03-29T11:40:26.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.170+0000 7efc7a52b640 10 monclient: shutdown 2024-03-29T11:40:26.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.170+0000 7efc7a52b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:26.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.170+0000 7efc7a52b640 1 -- 172.21.15.37:0/1818774703 >> v1:172.21.15.37:6790/0 conn(0x7efc740b1f60 legacy=0x7efc74143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.170+0000 7efc72ffd640 1 -- 172.21.15.37:0/1818774703 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7efc64001cc0 con 0x7efc740b1f60 2024-03-29T11:40:26.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.170+0000 7efc72ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:40:26.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.171+0000 7efc7a52b640 1 -- 172.21.15.37:0/1818774703 shutdown_connections 2024-03-29T11:40:26.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.171+0000 7efc7a52b640 1 -- 172.21.15.37:0/1818774703 >> 172.21.15.37:0/1818774703 conn(0x7efc74066a00 msgr2=0x7efc74066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:26.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.173+0000 7efc7a52b640 1 -- 172.21.15.37:0/1818774703 shutdown_connections 2024-03-29T11:40:26.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.173+0000 7efc7a52b640 1 -- 172.21.15.37:0/1818774703 wait complete. 2024-03-29T11:40:26.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.174+0000 7efc7a52b640 10 monclient: build_initial_monmap 2024-03-29T11:40:26.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.174+0000 7efc7a52b640 10 monclient: monmap: 2024-03-29T11:40:26.177 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:26.177 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:26.177 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:40:26.177 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:40:26.178 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:26.178 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:26.178 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:26.178 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:40:26.178 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:40:26.178 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:26.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.174+0000 7efc7a52b640 1 librados: starting msgr at 2024-03-29T11:40:26.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.174+0000 7efc7a52b640 1 librados: starting objecter 2024-03-29T11:40:26.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.174+0000 7efc7a52b640 1 Processor -- start 2024-03-29T11:40:26.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.178+0000 7efc7a52b640 1 -- start start 2024-03-29T11:40:26.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.178+0000 7efc7a52b640 1 librados: setting wanted keys 2024-03-29T11:40:26.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.178+0000 7efc7a52b640 1 librados: calling monclient init 2024-03-29T11:40:26.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.178+0000 7efc7a52b640 10 monclient: init 2024-03-29T11:40:26.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.179+0000 7efc7a52b640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:26.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.179+0000 7efc7a52b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:26.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.179+0000 7efc7a52b640 10 monclient(hunting): picked mon.noname-c con 0x7efc740b1f60 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.183+0000 7efc7a52b640 10 monclient(hunting): picked mon.noname-a con 0x7efc74143d10 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.186+0000 7efc737fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7efc74143d10 0x7efc74153770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41655/0 (socket says 172.21.15.37:41655) 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.186+0000 7efc78ac8640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7efc740aa6b0 0x7efc741ee670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:43275/0 (socket says 172.21.15.37:43275) 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.186+0000 7efc737fe640 1 -- 172.21.15.37:0/3293611222 learned_addr learned my addr 172.21.15.37:0/3293611222 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.186+0000 7efc70ff9640 10 client.?.objecter ms_handle_connect 0x7efc74143d10 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.186+0000 7efc70ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc7a52b640 10 monclient(hunting): picked mon.noname-b con 0x7efc740aa6b0 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc7a52b640 1 -- 172.21.15.37:0/3293611222 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc740ac270 con 0x7efc74143d10 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc7a52b640 1 -- 172.21.15.37:0/3293611222 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc740aab90 con 0x7efc740aa6b0 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc7a52b640 1 -- 172.21.15.37:0/3293611222 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc740aa320 con 0x7efc740b1f60 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690872551 0 0) 0x7efc740ac270 con 0x7efc74143d10 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc7a52b640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc7a52b640 10 monclient(hunting): authenticate will time out at 1205.636437s 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc70ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc70ff9640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc60004eb0 con 0x7efc74143d10 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc70ff9640 10 client.?.objecter ms_handle_connect 0x7efc740aa6b0 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc70ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357134722 0 0) 0x7efc740aab90 con 0x7efc740aa6b0 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc70ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc70ff9640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.187+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc740ac270 con 0x7efc740aa6b0 2024-03-29T11:40:26.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 10 client.?.objecter ms_handle_connect 0x7efc740b1f60 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2720615537 0 0) 0x7efc740ac270 con 0x7efc740aa6b0 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc740aab90 con 0x7efc740aa6b0 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7efc68001710 con 0x7efc740aa6b0 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 10 monclient(hunting): dump: 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802894321 0 0) 0x7efc740aa320 con 0x7efc740b1f60 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.188+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc740ac270 con 0x7efc740b1f60 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.189+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3194470084 0 0) 0x7efc60004eb0 con 0x7efc74143d10 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.189+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc740aa320 con 0x7efc74143d10 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.189+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7efc6c001dd0 con 0x7efc74143d10 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.189+0000 7efc70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.189+0000 7efc70ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:26.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.189+0000 7efc70ff9640 10 monclient(hunting): dump: 2024-03-29T11:40:26.196 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:26.196 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:26.196 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:26.196 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:26.196 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.196 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:26.196 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:26.196 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:26.196 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:26.196 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:26.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.191+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1216492619 0 0) 0x7efc740aa320 con 0x7efc74143d10 2024-03-29T11:40:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.191+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 >> v1:172.21.15.37:6790/0 conn(0x7efc740b1f60 legacy=0x7efc740a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.191+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 >> v1:172.21.15.146:6789/0 conn(0x7efc740aa6b0 legacy=0x7efc741ee670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.191+0000 7efc70ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:40:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.191+0000 7efc70ff9640 1 monclient: found mon.a 2024-03-29T11:40:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.191+0000 7efc70ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.191+0000 7efc70ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.191+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc740aa4b0 con 0x7efc74143d10 2024-03-29T11:40:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.191+0000 7efc70ff9640 10 monclient: _finish_auth 0 2024-03-29T11:40:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.191+0000 7efc70ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:40:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.191+0000 7efc70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.191+0000 7efc7a52b640 5 monclient: authenticate success, global_id 4416 2024-03-29T11:40:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.191+0000 7efc7a52b640 10 monclient: _renew_subs 2024-03-29T11:40:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.191+0000 7efc7a52b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.192+0000 7efc7a52b640 1 -- 172.21.15.37:0/3293611222 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc740ab090 con 0x7efc74143d10 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.192+0000 7efc7a52b640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.192+0000 7efc7a52b640 10 monclient: _renew_subs 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.192+0000 7efc7a52b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.192+0000 7efc7a52b640 1 -- 172.21.15.37:0/3293611222 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc74154060 con 0x7efc74143d10 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.192+0000 7efc7a52b640 1 librados: init done 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.194+0000 7efc7a52b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.194+0000 7efc7a52b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.194+0000 7efc7a52b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.194+0000 7efc7a52b640 1 -- 172.21.15.37:0/3293611222 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc740ab280 con 0x7efc74143d10 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.196+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc6c004b60 con 0x7efc74143d10 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.196+0000 7efc70ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.196+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7efc6c004e70 con 0x7efc74143d10 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.196+0000 7efc70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.196+0000 7efc70ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.196+0000 7efc70ff9640 10 monclient: dump: 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.196+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7efc6c01ae80 con 0x7efc74143d10 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.197+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7efc6c084e70 con 0x7efc74143d10 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.197+0000 7efc70ff9640 10 client.4416.objecter ms_dispatch 0x7efc74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.197+0000 7efc70ff9640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.197+0000 7efc70ff9640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:26.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.197+0000 7efc70ff9640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T11:40:26.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.197+0000 7efc727fc640 4 set_mon_vals no callback set 2024-03-29T11:40:26.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.198+0000 7efc73fff640 10 monclient: get_auth_request con 0x7efc6006dfb0 auth_method 0 2024-03-29T11:40:26.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.200+0000 7efc70ff9640 10 client.4416.objecter ms_handle_connect 0x7efc6006dfb0 2024-03-29T11:40:26.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.205+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc6c0508c0 con 0x7efc74143d10 2024-03-29T11:40:26.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.205+0000 7efc70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:26.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.205+0000 7efc70ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:26.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.535+0000 7efc7a52b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:26.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.535+0000 7efc7a52b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:26.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.535+0000 7efc7a52b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:26.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.535+0000 7efc7a52b640 1 -- 172.21.15.37:0/3293611222 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc74075f10 con 0x7efc74143d10 2024-03-29T11:40:26.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.536+0000 7efc70ff9640 1 -- 172.21.15.37:0/3293611222 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7efc6c0508c0 con 0x7efc74143d10 2024-03-29T11:40:26.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.536+0000 7efc70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:26.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.536+0000 7efc70ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:26.539 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:40:26.539 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:26.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.540+0000 7efc7a52b640 10 librados: watch_flush enter 2024-03-29T11:40:26.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.541+0000 7efc7a52b640 10 librados: watch_flush exit 2024-03-29T11:40:26.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.541+0000 7efc7a52b640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T11:40:26.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.541+0000 7efc7a52b640 10 client.4416.objecter successfully canceled tick 2024-03-29T11:40:26.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.542+0000 7efc7a52b640 1 -- 172.21.15.37:0/3293611222 >> v1:172.21.15.146:6816/2899149139 conn(0x7efc6006dfb0 legacy=0x7efc60070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.543+0000 7efc7a52b640 10 monclient: shutdown 2024-03-29T11:40:26.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.543+0000 7efc7a52b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:26.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.543+0000 7efc7a52b640 1 -- 172.21.15.37:0/3293611222 >> v1:172.21.15.37:6789/0 conn(0x7efc74143d10 legacy=0x7efc74153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.543+0000 7efc7a52b640 1 -- 172.21.15.37:0/3293611222 shutdown_connections 2024-03-29T11:40:26.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.543+0000 7efc7a52b640 1 -- 172.21.15.37:0/3293611222 >> 172.21.15.37:0/3293611222 conn(0x7efc74066a00 msgr2=0x7efc74066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:26.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.543+0000 7efc7a52b640 1 -- 172.21.15.37:0/3293611222 shutdown_connections 2024-03-29T11:40:26.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.543+0000 7efc7a52b640 1 -- 172.21.15.37:0/3293611222 wait complete. 2024-03-29T11:40:26.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:26.543+0000 7efc7a52b640 1 librados: shutdown 2024-03-29T11:40:26.561 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:33.369 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:33.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.516+0000 7f3051602640 10 monclient: get_monmap_and_config 2024-03-29T11:40:33.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.516+0000 7f3051602640 10 monclient: build_initial_monmap 2024-03-29T11:40:33.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.516+0000 7f3051602640 10 monclient: monmap: 2024-03-29T11:40:33.519 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:33.519 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:33.519 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:40:33.517666+0000 2024-03-29T11:40:33.519 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:40:33.517666+0000 2024-03-29T11:40:33.519 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:33.519 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:33.519 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:33.519 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:40:33.519 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:40:33.519 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:33.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.520+0000 7f3051602640 1 Processor -- start 2024-03-29T11:40:33.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.520+0000 7f3051602640 1 -- start start 2024-03-29T11:40:33.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.520+0000 7f3051602640 10 monclient: init 2024-03-29T11:40:33.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.521+0000 7f3051602640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:33.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.521+0000 7f3051602640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:33.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.521+0000 7f3051602640 10 monclient(hunting): picked mon.noname-a con 0x7f304c06cbb0 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:33.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.521+0000 7f3051602640 10 monclient(hunting): picked mon.noname-c con 0x7f304c072150 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:33.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.521+0000 7f3051602640 10 monclient(hunting): picked mon.noname-b con 0x7f304c06d6b0 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:33.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.521+0000 7f3051602640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c06ff70 con 0x7f304c06cbb0 2024-03-29T11:40:33.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.521+0000 7f3051602640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c06be10 con 0x7f304c072150 2024-03-29T11:40:33.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.521+0000 7f3051602640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c06d1c0 con 0x7f304c06d6b0 2024-03-29T11:40:33.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.521+0000 7f3051602640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:33.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.521+0000 7f3051602640 10 monclient(hunting): authenticate will time out at 1212.970454s 2024-03-29T11:40:33.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.522+0000 7f3050e01640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f304c06d6b0 0x7f304c146d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:35892/0 (socket says 172.21.15.37:35892) 2024-03-29T11:40:33.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.522+0000 7f3050e01640 1 -- 172.21.15.37:0/3297560780 learned_addr learned my addr 172.21.15.37:0/3297560780 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:33.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.522+0000 7f3043fff640 1 -- 172.21.15.37:0/3297560780 >> v1:172.21.15.37:6789/0 conn(0x7f304c06cbb0 legacy=0x7f304c071a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-29T11:40:33.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.522+0000 7f3043fff640 1 -- 172.21.15.37:0/3297560780 >> v1:172.21.15.37:6789/0 conn(0x7f304c06cbb0 legacy=0x7f304c071a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:40:33.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.522+0000 7f3043fff640 1 --1- 172.21.15.37:0/3297560780 >> v1:172.21.15.37:6789/0 conn(0x7f304c06cbb0 0x7f304c071a40 :-1 s=OPENED pgs=546489 cs=1 l=1).handle_message read tag failed 2024-03-29T11:40:33.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.522+0000 7f3043fff640 1 --1- 172.21.15.37:0/3297560780 >> v1:172.21.15.37:6789/0 conn(0x7f304c06cbb0 0x7f304c071a40 :-1 s=OPENED pgs=546489 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:40:33.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.523+0000 7f304b7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.37:6789/0 2024-03-29T11:40:33.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.523+0000 7f304b7fe640 1 -- 172.21.15.37:0/3297560780 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973236888 0 0) 0x7f304c06d1c0 con 0x7f304c06d6b0 2024-03-29T11:40:33.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.523+0000 7f304b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:33.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.523+0000 7f304b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:33.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.523+0000 7f304b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:33.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.523+0000 7f304b7fe640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:40:33.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.524+0000 7f304b7fe640 1 -- 172.21.15.37:0/3297560780 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3038003520 con 0x7f304c06d6b0 2024-03-29T11:40:33.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.524+0000 7f304b7fe640 1 -- 172.21.15.37:0/3297560780 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1783371075 0 0) 0x7f3038003520 con 0x7f304c06d6b0 2024-03-29T11:40:33.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.524+0000 7f304b7fe640 1 -- 172.21.15.37:0/3297560780 >> v1:172.21.15.146:6789/0 conn(0x7f304c072150 legacy=0x7f304c146670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:33.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f304b7fe640 1 -- 172.21.15.37:0/3297560780 >> v1:172.21.15.37:6789/0 conn(0x7f304c06cbb0 legacy=0x7f304c071a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f304b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f304b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f304b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f304b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f304b7fe640 1 -- 172.21.15.37:0/3297560780 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c155d40 con 0x7f304c06d6b0 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f304b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f304b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f304b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f304b7fe640 1 -- 172.21.15.37:0/3297560780 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3044001a10 con 0x7f304c06d6b0 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f304b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f304b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f304b7fe640 10 monclient: dump: 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:33.531 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:33.531 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:33.531 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:33.531 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:33.531 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:33.531 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:33.531 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:33.531 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:33.531 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f3051602640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:40:33.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.525+0000 7f3051602640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:33.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.526+0000 7f304b7fe640 1 -- 172.21.15.37:0/3297560780 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3044001c30 con 0x7f304c06d6b0 2024-03-29T11:40:33.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.526+0000 7f304b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:33.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.526+0000 7f304b7fe640 1 -- 172.21.15.37:0/3297560780 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3044001f40 con 0x7f304c06d6b0 2024-03-29T11:40:33.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.526+0000 7f3051602640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:33.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.526+0000 7f3051602640 4 set_mon_vals no callback set 2024-03-29T11:40:33.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.527+0000 7f3051602640 10 monclient: shutdown 2024-03-29T11:40:33.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.527+0000 7f3051602640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:33.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.527+0000 7f3051602640 1 -- 172.21.15.37:0/3297560780 >> v1:172.21.15.37:6790/0 conn(0x7f304c06d6b0 legacy=0x7f304c146d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:33.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.527+0000 7f304b7fe640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:40:33.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.527+0000 7f3051602640 1 -- 172.21.15.37:0/3297560780 shutdown_connections 2024-03-29T11:40:33.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.527+0000 7f3051602640 1 -- 172.21.15.37:0/3297560780 >> 172.21.15.37:0/3297560780 conn(0x7f304c0674d0 msgr2=0x7f304c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:33.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.528+0000 7f3051602640 1 -- 172.21.15.37:0/3297560780 shutdown_connections 2024-03-29T11:40:33.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.528+0000 7f3051602640 1 -- 172.21.15.37:0/3297560780 wait complete. 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.529+0000 7f3051602640 10 monclient: build_initial_monmap 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.529+0000 7f3051602640 10 monclient: monmap: 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.530+0000 7f3051602640 1 librados: starting msgr at 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.530+0000 7f3051602640 1 librados: starting objecter 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.530+0000 7f3051602640 1 Processor -- start 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.530+0000 7f3051602640 1 -- start start 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.530+0000 7f3051602640 1 librados: setting wanted keys 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.530+0000 7f3051602640 1 librados: calling monclient init 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.530+0000 7f3051602640 10 monclient: init 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.531+0000 7f3051602640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.531+0000 7f3051602640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.531+0000 7f3051602640 10 monclient(hunting): picked mon.noname-a con 0x7f304c06cbb0 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.533+0000 7f3051602640 10 monclient(hunting): picked mon.noname-b con 0x7f304c06d6b0 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.533+0000 7f3051602640 10 monclient(hunting): picked mon.noname-c con 0x7f304c072150 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.533+0000 7f3051602640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c1483b0 con 0x7f304c06cbb0 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.533+0000 7f3051602640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c149210 con 0x7f304c06d6b0 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.533+0000 7f3051602640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c13bd70 con 0x7f304c072150 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.533+0000 7f3051602640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.533+0000 7f3051602640 10 monclient(hunting): authenticate will time out at 1212.982520s 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.533+0000 7f3043fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f304c06cbb0 0x7f304c147ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:60434/0 (socket says 172.21.15.37:60434) 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.533+0000 7f3043fff640 1 -- 172.21.15.37:0/682784989 learned_addr learned my addr 172.21.15.37:0/682784989 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.534+0000 7f30497fa640 10 client.?.objecter ms_handle_connect 0x7f304c06cbb0 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.534+0000 7f30497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.534+0000 7f30497fa640 10 client.?.objecter ms_handle_connect 0x7f304c06d6b0 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.534+0000 7f30497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.534+0000 7f30497fa640 10 client.?.objecter ms_handle_connect 0x7f304c072150 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.534+0000 7f30497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.534+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665381185 0 0) 0x7f304c149210 con 0x7f304c06d6b0 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.534+0000 7f30497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.534+0000 7f30497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.534+0000 7f30497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.534+0000 7f30497fa640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.535+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f302c004eb0 con 0x7f304c06d6b0 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.535+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954812524 0 0) 0x7f304c1483b0 con 0x7f304c06cbb0 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.535+0000 7f30497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.535+0000 7f30497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.535+0000 7f30497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.535+0000 7f30497fa640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.535+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f304c149210 con 0x7f304c06cbb0 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.535+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85600237 0 0) 0x7f304c13bd70 con 0x7f304c072150 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f304c1483b0 con 0x7f304c072150 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2141887086 0 0) 0x7f304c149210 con 0x7f304c06cbb0 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f304c13bd70 con 0x7f304c06cbb0 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2675625177 0 0) 0x7f302c004eb0 con 0x7f304c06d6b0 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f304c149210 con 0x7f304c06d6b0 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f303c0018b0 con 0x7f304c06cbb0 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 10 monclient(hunting): dump: 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3034001750 con 0x7f304c06d6b0 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:33.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:33.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.536+0000 7f30497fa640 10 monclient(hunting): dump: 2024-03-29T11:40:33.542 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:33.542 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:33.542 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:33.542 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:33.542 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:33.542 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:33.542 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:33.542 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:33.542 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:33.542 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1286721527 0 0) 0x7f304c149210 con 0x7f304c06d6b0 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 >> v1:172.21.15.37:6790/0 conn(0x7f304c072150 legacy=0x7f304c1e6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 >> v1:172.21.15.37:6789/0 conn(0x7f304c06cbb0 legacy=0x7f304c147ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 10 monclient: _finish_hunting 0 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 1 monclient: found mon.b 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c0a94e0 con 0x7f304c06d6b0 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 10 monclient: _finish_auth 0 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 10 monclient: _check_auth_tickets 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f3051602640 5 monclient: authenticate success, global_id 4381 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f3051602640 10 monclient: _renew_subs 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f3051602640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f3051602640 1 -- 172.21.15.37:0/682784989 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f304c0a96f0 con 0x7f304c06d6b0 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f3051602640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f3051602640 10 monclient: _renew_subs 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f3051602640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f3051602640 1 -- 172.21.15.37:0/682784989 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f304c13cbe0 con 0x7f304c06d6b0 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f3051602640 1 librados: init done 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30340018b0 con 0x7f304c06d6b0 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f3034001d60 con 0x7f304c06d6b0 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.537+0000 7f30497fa640 10 monclient: dump: 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.538+0000 7f304affd640 4 set_mon_vals no callback set 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.539+0000 7f3051602640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.539+0000 7f3051602640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.539+0000 7f3051602640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.539+0000 7f3051602640 1 -- 172.21.15.37:0/682784989 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3030005080 con 0x7f304c06d6b0 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.540+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f3034019600 con 0x7f304c06d6b0 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.540+0000 7f3043fff640 10 monclient: get_auth_request con 0x7f302c06dde0 auth_method 0 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.541+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f3034083640 con 0x7f304c06d6b0 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.541+0000 7f30497fa640 10 client.4381.objecter ms_dispatch 0x7f304c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.541+0000 7f30497fa640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.541+0000 7f30497fa640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.541+0000 7f30497fa640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.541+0000 7f30497fa640 10 client.4381.objecter ms_handle_connect 0x7f302c06dde0 2024-03-29T11:40:33.550 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.548+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f303404ed50 con 0x7f304c06d6b0 2024-03-29T11:40:33.550 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.548+0000 7f30497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:33.550 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.548+0000 7f30497fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:33.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.901+0000 7f3051602640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:33.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.901+0000 7f3051602640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:33.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.901+0000 7f3051602640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:40:33.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.901+0000 7f3051602640 1 -- 172.21.15.37:0/682784989 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3030005330 con 0x7f304c06d6b0 2024-03-29T11:40:33.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.902+0000 7f30497fa640 1 -- 172.21.15.37:0/682784989 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f3034050e50 con 0x7f304c06d6b0 2024-03-29T11:40:33.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.902+0000 7f30497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:33.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.902+0000 7f30497fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:33.907 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:40:33.907 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:33.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.904+0000 7f3051602640 10 librados: watch_flush enter 2024-03-29T11:40:33.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.905+0000 7f3051602640 10 librados: watch_flush exit 2024-03-29T11:40:33.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.905+0000 7f3051602640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-29T11:40:33.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.905+0000 7f3051602640 10 client.4381.objecter successfully canceled tick 2024-03-29T11:40:33.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.907+0000 7f3051602640 1 -- 172.21.15.37:0/682784989 >> v1:172.21.15.146:6816/2899149139 conn(0x7f302c06dde0 legacy=0x7f302c0702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:33.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.907+0000 7f3051602640 10 monclient: shutdown 2024-03-29T11:40:33.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.907+0000 7f3051602640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:33.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.907+0000 7f3051602640 1 -- 172.21.15.37:0/682784989 >> v1:172.21.15.146:6789/0 conn(0x7f304c06d6b0 legacy=0x7f304c13c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:33.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.908+0000 7f3051602640 1 -- 172.21.15.37:0/682784989 shutdown_connections 2024-03-29T11:40:33.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.908+0000 7f3051602640 1 -- 172.21.15.37:0/682784989 >> 172.21.15.37:0/682784989 conn(0x7f304c0674d0 msgr2=0x7f304c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:33.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.908+0000 7f3051602640 1 -- 172.21.15.37:0/682784989 shutdown_connections 2024-03-29T11:40:33.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.908+0000 7f3051602640 1 -- 172.21.15.37:0/682784989 wait complete. 2024-03-29T11:40:33.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:33.908+0000 7f3051602640 1 librados: shutdown 2024-03-29T11:40:33.926 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:40.734 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.873+0000 7fbc4ad23640 10 monclient: get_monmap_and_config 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.873+0000 7fbc4ad23640 10 monclient: build_initial_monmap 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.873+0000 7fbc4ad23640 10 monclient: monmap: 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:40:40.874343+0000 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:40:40.874343+0000 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.874+0000 7fbc4ad23640 1 Processor -- start 2024-03-29T11:40:40.877 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.874+0000 7fbc4ad23640 1 -- start start 2024-03-29T11:40:40.877 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.874+0000 7fbc4ad23640 10 monclient: init 2024-03-29T11:40:40.877 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.875+0000 7fbc4ad23640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:40.877 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.875+0000 7fbc4ad23640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:40.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.876+0000 7fbc4ad23640 10 monclient(hunting): picked mon.noname-b con 0x7fbc441422b0 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:40.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.876+0000 7fbc4ad23640 10 monclient(hunting): picked mon.noname-c con 0x7fbc44142690 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:40.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.876+0000 7fbc4ad23640 10 monclient(hunting): picked mon.noname-a con 0x7fbc440b17d0 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:40.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.876+0000 7fbc4ad23640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc44142c80 con 0x7fbc440b17d0 2024-03-29T11:40:40.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.876+0000 7fbc4ad23640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc440aae40 con 0x7fbc44142690 2024-03-29T11:40:40.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.876+0000 7fbc4ad23640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4413fbd0 con 0x7fbc441422b0 2024-03-29T11:40:40.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.876+0000 7fbc4ad23640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:40.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.876+0000 7fbc4ad23640 10 monclient(hunting): authenticate will time out at 1220.325580s 2024-03-29T11:40:40.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.876+0000 7fbc43fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbc44142690 0x7fbc440b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:34572/0 (socket says 172.21.15.37:34572) 2024-03-29T11:40:40.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.876+0000 7fbc43fff640 1 -- 172.21.15.37:0/3983720821 learned_addr learned my addr 172.21.15.37:0/3983720821 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:40.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.878+0000 7fbc437fe640 1 -- 172.21.15.37:0/3983720821 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991261466 0 0) 0x7fbc440aae40 con 0x7fbc44142690 2024-03-29T11:40:40.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.878+0000 7fbc437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:40.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.878+0000 7fbc437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.878+0000 7fbc437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.878+0000 7fbc437fe640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.878+0000 7fbc437fe640 1 -- 172.21.15.37:0/3983720821 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc34003520 con 0x7fbc44142690 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.878+0000 7fbc437fe640 1 -- 172.21.15.37:0/3983720821 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626265133 0 0) 0x7fbc44142c80 con 0x7fbc440b17d0 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.878+0000 7fbc437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.878+0000 7fbc437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.878+0000 7fbc437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.878+0000 7fbc437fe640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.878+0000 7fbc437fe640 1 -- 172.21.15.37:0/3983720821 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc440aae40 con 0x7fbc440b17d0 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.878+0000 7fbc437fe640 1 -- 172.21.15.37:0/3983720821 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4003429443 0 0) 0x7fbc440aae40 con 0x7fbc440b17d0 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.878+0000 7fbc437fe640 1 -- 172.21.15.37:0/3983720821 >> v1:172.21.15.37:6790/0 conn(0x7fbc441422b0 legacy=0x7fbc4413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 1 -- 172.21.15.37:0/3983720821 >> v1:172.21.15.146:6789/0 conn(0x7fbc44142690 legacy=0x7fbc440b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 1 monclient: found mon.noname-a 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 1 -- 172.21.15.37:0/3983720821 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc44151940 con 0x7fbc440b17d0 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 1 -- 172.21.15.37:0/3983720821 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbc3c0018a0 con 0x7fbc440b17d0 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 10 monclient: dump: 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc4ad23640 5 monclient: authenticate success, global_id 4422 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc4ad23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 1 -- 172.21.15.37:0/3983720821 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc3c001a00 con 0x7fbc440b17d0 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 1 -- 172.21.15.37:0/3983720821 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbc3c001d90 con 0x7fbc440b17d0 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc437fe640 10 monclient: dump: 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc4ad23640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.879+0000 7fbc4ad23640 4 set_mon_vals no callback set 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.880+0000 7fbc4ad23640 10 monclient: shutdown 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.880+0000 7fbc4ad23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.880+0000 7fbc4ad23640 1 -- 172.21.15.37:0/3983720821 >> v1:172.21.15.37:6789/0 conn(0x7fbc440b17d0 legacy=0x7fbc440b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.880+0000 7fbc4ad23640 1 -- 172.21.15.37:0/3983720821 shutdown_connections 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.880+0000 7fbc4ad23640 1 -- 172.21.15.37:0/3983720821 >> 172.21.15.37:0/3983720821 conn(0x7fbc44066a00 msgr2=0x7fbc44066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.881+0000 7fbc4ad23640 1 -- 172.21.15.37:0/3983720821 shutdown_connections 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.881+0000 7fbc4ad23640 1 -- 172.21.15.37:0/3983720821 wait complete. 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.882+0000 7fbc4ad23640 10 monclient: build_initial_monmap 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.882+0000 7fbc4ad23640 10 monclient: monmap: 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.882+0000 7fbc4ad23640 1 librados: starting msgr at 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.882+0000 7fbc4ad23640 1 librados: starting objecter 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.883+0000 7fbc4ad23640 1 Processor -- start 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.883+0000 7fbc4ad23640 1 -- start start 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.883+0000 7fbc4ad23640 1 librados: setting wanted keys 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.883+0000 7fbc4ad23640 1 librados: calling monclient init 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.883+0000 7fbc4ad23640 10 monclient: init 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.884+0000 7fbc4ad23640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.884+0000 7fbc4ad23640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.885+0000 7fbc4ad23640 10 monclient(hunting): picked mon.noname-a con 0x7fbc440b17d0 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.885+0000 7fbc4ad23640 10 monclient(hunting): picked mon.noname-b con 0x7fbc441422b0 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.885+0000 7fbc4ad23640 10 monclient(hunting): picked mon.noname-c con 0x7fbc44142690 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.885+0000 7fbc4ad23640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc440b7160 con 0x7fbc440b17d0 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.885+0000 7fbc4ad23640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4412dbb0 con 0x7fbc441422b0 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.885+0000 7fbc4ad23640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc440ac650 con 0x7fbc44142690 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.885+0000 7fbc4ad23640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.885+0000 7fbc4ad23640 10 monclient(hunting): authenticate will time out at 1220.334227s 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.886+0000 7fbc43fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbc441422b0 0x7fbc4412cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:40338/0 (socket says 172.21.15.37:40338) 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.886+0000 7fbc43fff640 1 -- 172.21.15.37:0/4223600756 learned_addr learned my addr 172.21.15.37:0/4223600756 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:40.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.886+0000 7fbc417fa640 10 client.?.objecter ms_handle_connect 0x7fbc441422b0 2024-03-29T11:40:40.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.886+0000 7fbc417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:40.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.886+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401647255 0 0) 0x7fbc4412dbb0 con 0x7fbc441422b0 2024-03-29T11:40:40.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.886+0000 7fbc417fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:40.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.886+0000 7fbc417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:40.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.886+0000 7fbc417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:40.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.886+0000 7fbc417fa640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:40:40.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.886+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc28004eb0 con 0x7fbc441422b0 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.887+0000 7fbc417fa640 10 client.?.objecter ms_handle_connect 0x7fbc440b17d0 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.887+0000 7fbc417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.888+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466028521 0 0) 0x7fbc440b7160 con 0x7fbc440b17d0 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.888+0000 7fbc417fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.888+0000 7fbc417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.888+0000 7fbc417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.888+0000 7fbc417fa640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.888+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc4412dbb0 con 0x7fbc440b17d0 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1801615793 0 0) 0x7fbc4412dbb0 con 0x7fbc440b17d0 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc440b7160 con 0x7fbc440b17d0 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbc38001830 con 0x7fbc440b17d0 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 10 monclient(hunting): dump: 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 10 client.?.objecter ms_handle_connect 0x7fbc44142690 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735620865 0 0) 0x7fbc440ac650 con 0x7fbc44142690 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc4412dbb0 con 0x7fbc44142690 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 169373128 0 0) 0x7fbc440b7160 con 0x7fbc440b17d0 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 >> v1:172.21.15.37:6790/0 conn(0x7fbc44142690 legacy=0x7fbc4412d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.889+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 >> v1:172.21.15.146:6789/0 conn(0x7fbc441422b0 legacy=0x7fbc4412cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc417fa640 10 monclient: _finish_hunting 0 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc417fa640 1 monclient: found mon.a 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc417fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc440ac930 con 0x7fbc440b17d0 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc417fa640 10 monclient: _finish_auth 0 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc417fa640 10 monclient: _check_auth_tickets 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc4ad23640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc4ad23640 10 monclient: _renew_subs 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc4ad23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc4ad23640 1 -- 172.21.15.37:0/4223600756 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4412e750 con 0x7fbc440b17d0 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc4ad23640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc4ad23640 10 monclient: _renew_subs 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc4ad23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc4ad23640 1 -- 172.21.15.37:0/4223600756 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc440b6bb0 con 0x7fbc440b17d0 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.890+0000 7fbc4ad23640 1 librados: init done 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.891+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc38001d40 con 0x7fbc440b17d0 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.891+0000 7fbc417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.891+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbc38003790 con 0x7fbc440b17d0 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.891+0000 7fbc417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.891+0000 7fbc417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.891+0000 7fbc42ffd640 4 set_mon_vals no callback set 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.891+0000 7fbc417fa640 10 monclient: dump: 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.893+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fbc38019760 con 0x7fbc440b17d0 2024-03-29T11:40:40.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.893+0000 7fbc4ad23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:40.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.893+0000 7fbc4ad23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:40.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.893+0000 7fbc4ad23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:40.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.893+0000 7fbc4ad23640 1 -- 172.21.15.37:0/4223600756 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc24005080 con 0x7fbc440b17d0 2024-03-29T11:40:40.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.894+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fbc380837a0 con 0x7fbc440b17d0 2024-03-29T11:40:40.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.894+0000 7fbc417fa640 10 client.4425.objecter ms_dispatch 0x7fbc44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:40.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.894+0000 7fbc417fa640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:40.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.894+0000 7fbc417fa640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:40.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.894+0000 7fbc417fa640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-29T11:40:40.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.894+0000 7fbc43fff640 10 monclient: get_auth_request con 0x7fbc2806de80 auth_method 0 2024-03-29T11:40:40.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.894+0000 7fbc417fa640 10 client.4425.objecter ms_handle_connect 0x7fbc2806de80 2024-03-29T11:40:40.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.901+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc3804f080 con 0x7fbc440b17d0 2024-03-29T11:40:40.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.901+0000 7fbc417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:40.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:40.901+0000 7fbc417fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:41.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.239+0000 7fbc4ad23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:41.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.239+0000 7fbc4ad23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:41.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.239+0000 7fbc4ad23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:40:41.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.239+0000 7fbc4ad23640 1 -- 172.21.15.37:0/4223600756 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc24005330 con 0x7fbc440b17d0 2024-03-29T11:40:41.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.240+0000 7fbc417fa640 1 -- 172.21.15.37:0/4223600756 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fbc38050fb0 con 0x7fbc440b17d0 2024-03-29T11:40:41.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.240+0000 7fbc417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:41.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.240+0000 7fbc417fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:41.242 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:40:41.242 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:41.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.243+0000 7fbc4ad23640 10 librados: watch_flush enter 2024-03-29T11:40:41.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.244+0000 7fbc4ad23640 10 librados: watch_flush exit 2024-03-29T11:40:41.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.244+0000 7fbc4ad23640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-29T11:40:41.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.244+0000 7fbc4ad23640 10 client.4425.objecter successfully canceled tick 2024-03-29T11:40:41.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.244+0000 7fbc4ad23640 1 -- 172.21.15.37:0/4223600756 >> v1:172.21.15.146:6816/2899149139 conn(0x7fbc2806de80 legacy=0x7fbc28070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:41.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.244+0000 7fbc4ad23640 10 monclient: shutdown 2024-03-29T11:40:41.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.244+0000 7fbc4ad23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:41.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.244+0000 7fbc4ad23640 1 -- 172.21.15.37:0/4223600756 >> v1:172.21.15.37:6789/0 conn(0x7fbc440b17d0 legacy=0x7fbc4412e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:41.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.247+0000 7fbc4ad23640 1 -- 172.21.15.37:0/4223600756 shutdown_connections 2024-03-29T11:40:41.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.247+0000 7fbc4ad23640 1 -- 172.21.15.37:0/4223600756 >> 172.21.15.37:0/4223600756 conn(0x7fbc44066a00 msgr2=0x7fbc4413c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:41.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.247+0000 7fbc4ad23640 1 -- 172.21.15.37:0/4223600756 shutdown_connections 2024-03-29T11:40:41.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.247+0000 7fbc4ad23640 1 -- 172.21.15.37:0/4223600756 wait complete. 2024-03-29T11:40:41.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:41.247+0000 7fbc4ad23640 1 librados: shutdown 2024-03-29T11:40:41.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:48.075 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:48.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.207+0000 7fd413da5640 10 monclient: get_monmap_and_config 2024-03-29T11:40:48.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.207+0000 7fd413da5640 10 monclient: build_initial_monmap 2024-03-29T11:40:48.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.207+0000 7fd413da5640 10 monclient: monmap: 2024-03-29T11:40:48.209 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:48.209 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:48.209 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:40:48.208117+0000 2024-03-29T11:40:48.209 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:40:48.208117+0000 2024-03-29T11:40:48.209 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:48.209 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:48.210 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:48.210 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:40:48.210 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:40:48.210 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:48.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.208+0000 7fd413da5640 1 Processor -- start 2024-03-29T11:40:48.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.208+0000 7fd413da5640 1 -- start start 2024-03-29T11:40:48.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.208+0000 7fd413da5640 10 monclient: init 2024-03-29T11:40:48.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.209+0000 7fd413da5640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:48.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.209+0000 7fd413da5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:48.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.209+0000 7fd413da5640 10 monclient(hunting): picked mon.noname-a con 0x7fd40c0b1f60 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:48.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.209+0000 7fd413da5640 10 monclient(hunting): picked mon.noname-b con 0x7fd40c071eb0 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:48.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.209+0000 7fd413da5640 10 monclient(hunting): picked mon.noname-c con 0x7fd40c143d10 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:48.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.209+0000 7fd413da5640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c06ef80 con 0x7fd40c0b1f60 2024-03-29T11:40:48.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.209+0000 7fd413da5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c06f260 con 0x7fd40c143d10 2024-03-29T11:40:48.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.209+0000 7fd413da5640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c0b40c0 con 0x7fd40c071eb0 2024-03-29T11:40:48.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.209+0000 7fd413da5640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:48.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.209+0000 7fd413da5640 10 monclient(hunting): authenticate will time out at 1227.658536s 2024-03-29T11:40:48.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.209+0000 7fd411340640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fd40c071eb0 0x7fd40c146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:40402/0 (socket says 172.21.15.37:40402) 2024-03-29T11:40:48.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.209+0000 7fd411340640 1 -- 172.21.15.37:0/4113412209 learned_addr learned my addr 172.21.15.37:0/4113412209 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:48.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.209+0000 7fd410b3f640 1 -- 172.21.15.37:0/4113412209 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136808937 0 0) 0x7fd40c0b40c0 con 0x7fd40c071eb0 2024-03-29T11:40:48.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.210+0000 7fd410b3f640 10 monclient(hunting): _negotiate 2024-03-29T11:40:48.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.210+0000 7fd410b3f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:48.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.210+0000 7fd410b3f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:48.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.210+0000 7fd410b3f640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:40:48.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.210+0000 7fd410b3f640 1 -- 172.21.15.37:0/4113412209 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f8003520 con 0x7fd40c071eb0 2024-03-29T11:40:48.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.210+0000 7fd410b3f640 1 -- 172.21.15.37:0/4113412209 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879875234 0 0) 0x7fd40c06f260 con 0x7fd40c143d10 2024-03-29T11:40:48.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.210+0000 7fd410b3f640 10 monclient(hunting): _negotiate 2024-03-29T11:40:48.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.210+0000 7fd410b3f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:48.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.210+0000 7fd410b3f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:48.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.210+0000 7fd410b3f640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:40:48.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.210+0000 7fd410b3f640 1 -- 172.21.15.37:0/4113412209 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd40c0b40c0 con 0x7fd40c143d10 2024-03-29T11:40:48.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.210+0000 7fd410b3f640 1 -- 172.21.15.37:0/4113412209 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 379440414 0 0) 0x7fd3f8003520 con 0x7fd40c071eb0 2024-03-29T11:40:48.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.210+0000 7fd410b3f640 1 -- 172.21.15.37:0/4113412209 >> v1:172.21.15.146:6789/0 conn(0x7fd40c143d10 legacy=0x7fd40c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.210+0000 7fd410b3f640 1 -- 172.21.15.37:0/4113412209 >> v1:172.21.15.37:6789/0 conn(0x7fd40c0b1f60 legacy=0x7fd40c143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:48.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd410b3f640 10 monclient: _finish_hunting 0 2024-03-29T11:40:48.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd410b3f640 1 monclient: found mon.noname-b 2024-03-29T11:40:48.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd410b3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:48.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd410b3f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:40:48.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd410b3f640 1 -- 172.21.15.37:0/4113412209 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c155d70 con 0x7fd40c071eb0 2024-03-29T11:40:48.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd410b3f640 10 monclient: _finish_auth 0 2024-03-29T11:40:48.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd410b3f640 10 monclient: _check_auth_tickets 2024-03-29T11:40:48.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd410b3f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:48.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd410b3f640 1 -- 172.21.15.37:0/4113412209 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fd3fc001690 con 0x7fd40c071eb0 2024-03-29T11:40:48.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd410b3f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd410b3f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:48.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd410b3f640 10 monclient: dump: 2024-03-29T11:40:48.219 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd413da5640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd413da5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd410b3f640 1 -- 172.21.15.37:0/4113412209 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3fc001870 con 0x7fd40c071eb0 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd410b3f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd413da5640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:48.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.211+0000 7fd413da5640 4 set_mon_vals no callback set 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.212+0000 7fd410b3f640 1 -- 172.21.15.37:0/4113412209 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fd3fc001c80 con 0x7fd40c071eb0 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.212+0000 7fd410b3f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.212+0000 7fd410b3f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.212+0000 7fd410b3f640 10 monclient: dump: 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:48.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.212+0000 7fd413da5640 10 monclient: shutdown 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.212+0000 7fd413da5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.212+0000 7fd413da5640 1 -- 172.21.15.37:0/4113412209 >> v1:172.21.15.37:6790/0 conn(0x7fd40c071eb0 legacy=0x7fd40c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.213+0000 7fd413da5640 1 -- 172.21.15.37:0/4113412209 shutdown_connections 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.213+0000 7fd413da5640 1 -- 172.21.15.37:0/4113412209 >> 172.21.15.37:0/4113412209 conn(0x7fd40c066a00 msgr2=0x7fd40c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.213+0000 7fd413da5640 1 -- 172.21.15.37:0/4113412209 shutdown_connections 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.213+0000 7fd413da5640 1 -- 172.21.15.37:0/4113412209 wait complete. 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.214+0000 7fd413da5640 10 monclient: build_initial_monmap 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.214+0000 7fd413da5640 10 monclient: monmap: 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:48.222 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.214+0000 7fd413da5640 1 librados: starting msgr at 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.214+0000 7fd413da5640 1 librados: starting objecter 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.214+0000 7fd413da5640 1 Processor -- start 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.214+0000 7fd413da5640 1 -- start start 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.214+0000 7fd413da5640 1 librados: setting wanted keys 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.214+0000 7fd413da5640 1 librados: calling monclient init 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.214+0000 7fd413da5640 10 monclient: init 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.215+0000 7fd413da5640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.215+0000 7fd413da5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.216+0000 7fd413da5640 10 monclient(hunting): picked mon.noname-b con 0x7fd40c0b1f60 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.216+0000 7fd413da5640 10 monclient(hunting): picked mon.noname-c con 0x7fd40c143d10 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.216+0000 7fd411b41640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd40c0b1f60 0x7fd40c1390c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:35688/0 (socket says 172.21.15.37:35688) 2024-03-29T11:40:48.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.216+0000 7fd411b41640 1 -- 172.21.15.37:0/409675057 learned_addr learned my addr 172.21.15.37:0/409675057 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:48.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.216+0000 7fd413da5640 10 monclient(hunting): picked mon.noname-a con 0x7fd40c1397d0 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:48.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.216+0000 7fd413da5640 1 -- 172.21.15.37:0/409675057 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c0acdc0 con 0x7fd40c1397d0 2024-03-29T11:40:48.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.216+0000 7fd4027fc640 10 client.?.objecter ms_handle_connect 0x7fd40c0b1f60 2024-03-29T11:40:48.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.216+0000 7fd4027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:48.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.216+0000 7fd4027fc640 10 client.?.objecter ms_handle_connect 0x7fd40c143d10 2024-03-29T11:40:48.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.217+0000 7fd4027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:48.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.217+0000 7fd413da5640 1 -- 172.21.15.37:0/409675057 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c0ab620 con 0x7fd40c0b1f60 2024-03-29T11:40:48.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.217+0000 7fd413da5640 1 -- 172.21.15.37:0/409675057 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c136950 con 0x7fd40c143d10 2024-03-29T11:40:48.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.217+0000 7fd413da5640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:48.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.217+0000 7fd413da5640 10 monclient(hunting): authenticate will time out at 1227.666333s 2024-03-29T11:40:48.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.217+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551590679 0 0) 0x7fd40c0ab620 con 0x7fd40c0b1f60 2024-03-29T11:40:48.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.217+0000 7fd4027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:48.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.217+0000 7fd4027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:48.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.217+0000 7fd4027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.217+0000 7fd4027fc640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.218+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f0004eb0 con 0x7fd40c0b1f60 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.218+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484704314 0 0) 0x7fd40c136950 con 0x7fd40c143d10 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.218+0000 7fd4027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.218+0000 7fd4027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.218+0000 7fd4027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.218+0000 7fd4027fc640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.218+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd40c0ab620 con 0x7fd40c143d10 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.218+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2031929801 0 0) 0x7fd3f0004eb0 con 0x7fd40c0b1f60 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd40c136950 con 0x7fd40c0b1f60 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fd408001940 con 0x7fd40c0b1f60 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 10 monclient(hunting): dump: 2024-03-29T11:40:48.231 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1658520696 0 0) 0x7fd40c0ab620 con 0x7fd40c143d10 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3f0004eb0 con 0x7fd40c143d10 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fd3fc001b80 con 0x7fd40c143d10 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 10 monclient(hunting): dump: 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 10 client.?.objecter ms_handle_connect 0x7fd40c1397d0 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 886708427 0 0) 0x7fd40c136950 con 0x7fd40c0b1f60 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 >> v1:172.21.15.37:6790/0 conn(0x7fd40c143d10 legacy=0x7fd40c1dac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 >> v1:172.21.15.37:6789/0 conn(0x7fd40c1397d0 legacy=0x7fd40c1de340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 1 monclient: found mon.b 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c0aa6f0 con 0x7fd40c0b1f60 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd4027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd413da5640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd413da5640 10 monclient: _renew_subs 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd413da5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd413da5640 1 -- 172.21.15.37:0/409675057 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40c0abb40 con 0x7fd40c0b1f60 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd413da5640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:48.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd413da5640 10 monclient: _renew_subs 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd413da5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd413da5640 1 -- 172.21.15.37:0/409675057 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40c1545c0 con 0x7fd40c0b1f60 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.219+0000 7fd413da5640 1 librados: init done 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.220+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd408001aa0 con 0x7fd40c0b1f60 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.220+0000 7fd4027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.220+0000 7fd403fff640 4 set_mon_vals no callback set 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.220+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fd4080038a0 con 0x7fd40c0b1f60 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.220+0000 7fd4027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.220+0000 7fd4027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.220+0000 7fd4027fc640 10 monclient: dump: 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:48.235 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.221+0000 7fd413da5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.221+0000 7fd413da5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.221+0000 7fd413da5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.221+0000 7fd413da5640 1 -- 172.21.15.37:0/409675057 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f4005080 con 0x7fd40c0b1f60 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.222+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fd408003a80 con 0x7fd40c0b1f60 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.222+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fd408084a90 con 0x7fd40c0b1f60 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.222+0000 7fd4027fc640 10 client.4393.objecter ms_dispatch 0x7fd40c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.222+0000 7fd4027fc640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.222+0000 7fd4027fc640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:48.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.223+0000 7fd4027fc640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-29T11:40:48.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.223+0000 7fd411340640 10 monclient: get_auth_request con 0x7fd3f006de40 auth_method 0 2024-03-29T11:40:48.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.223+0000 7fd4027fc640 10 client.4393.objecter ms_handle_connect 0x7fd3f006de40 2024-03-29T11:40:48.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.232+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd408050370 con 0x7fd40c0b1f60 2024-03-29T11:40:48.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.232+0000 7fd4027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:48.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.232+0000 7fd4027fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:48.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.540+0000 7fd413da5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:48.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.540+0000 7fd413da5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:48.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.540+0000 7fd413da5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:40:48.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.540+0000 7fd413da5640 1 -- 172.21.15.37:0/409675057 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3f4005330 con 0x7fd40c0b1f60 2024-03-29T11:40:48.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.541+0000 7fd4027fc640 1 -- 172.21.15.37:0/409675057 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fd4080522a0 con 0x7fd40c0b1f60 2024-03-29T11:40:48.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.541+0000 7fd4027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:48.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.541+0000 7fd4027fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:48.544 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:40:48.544 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:48.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.545+0000 7fd413da5640 10 librados: watch_flush enter 2024-03-29T11:40:48.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.545+0000 7fd413da5640 10 librados: watch_flush exit 2024-03-29T11:40:48.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.545+0000 7fd413da5640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-29T11:40:48.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.545+0000 7fd413da5640 10 client.4393.objecter successfully canceled tick 2024-03-29T11:40:48.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.545+0000 7fd413da5640 1 -- 172.21.15.37:0/409675057 >> v1:172.21.15.146:6816/2899149139 conn(0x7fd3f006de40 legacy=0x7fd3f0070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.545+0000 7fd413da5640 10 monclient: shutdown 2024-03-29T11:40:48.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.545+0000 7fd413da5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:48.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.545+0000 7fd413da5640 1 -- 172.21.15.37:0/409675057 >> v1:172.21.15.146:6789/0 conn(0x7fd40c0b1f60 legacy=0x7fd40c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.545+0000 7fd413da5640 1 -- 172.21.15.37:0/409675057 shutdown_connections 2024-03-29T11:40:48.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.545+0000 7fd413da5640 1 -- 172.21.15.37:0/409675057 >> 172.21.15.37:0/409675057 conn(0x7fd40c066a00 msgr2=0x7fd40c0b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:48.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.545+0000 7fd413da5640 1 -- 172.21.15.37:0/409675057 shutdown_connections 2024-03-29T11:40:48.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.545+0000 7fd413da5640 1 -- 172.21.15.37:0/409675057 wait complete. 2024-03-29T11:40:48.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:48.545+0000 7fd413da5640 1 librados: shutdown 2024-03-29T11:40:48.560 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:55.367 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:55.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.521+0000 7f227d14d640 10 monclient: get_monmap_and_config 2024-03-29T11:40:55.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.521+0000 7f227d14d640 10 monclient: build_initial_monmap 2024-03-29T11:40:55.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.521+0000 7f227d14d640 10 monclient: monmap: 2024-03-29T11:40:55.524 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:55.524 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:55.524 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:40:55.522266+0000 2024-03-29T11:40:55.524 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:40:55.522266+0000 2024-03-29T11:40:55.524 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:55.524 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:55.524 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:55.524 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:40:55.524 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:40:55.524 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:55.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.522+0000 7f227d14d640 1 Processor -- start 2024-03-29T11:40:55.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.522+0000 7f227d14d640 1 -- start start 2024-03-29T11:40:55.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.522+0000 7f227d14d640 10 monclient: init 2024-03-29T11:40:55.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f227d14d640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:55.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f227d14d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:55.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f227d14d640 10 monclient(hunting): picked mon.noname-b con 0x7f22781457f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:55.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f227d14d640 10 monclient(hunting): picked mon.noname-c con 0x7f227813f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:55.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f227d14d640 10 monclient(hunting): picked mon.noname-a con 0x7f22780a9d70 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:55.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f227d14d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22780aaef0 con 0x7f22780a9d70 2024-03-29T11:40:55.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f227d14d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2278145bd0 con 0x7f227813f650 2024-03-29T11:40:55.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f227d14d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2278145eb0 con 0x7f22781457f0 2024-03-29T11:40:55.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f227d14d640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:55.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f227d14d640 10 monclient(hunting): authenticate will time out at 1234.972284s 2024-03-29T11:40:55.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f2276d9d640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f22781457f0 0x7f227813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:51348/0 (socket says 172.21.15.37:51348) 2024-03-29T11:40:55.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f2276d9d640 1 -- 172.21.15.37:0/4272711087 learned_addr learned my addr 172.21.15.37:0/4272711087 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:55.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f2275d9b640 1 -- 172.21.15.37:0/4272711087 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908038556 0 0) 0x7f2278145eb0 con 0x7f22781457f0 2024-03-29T11:40:55.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f2275d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:40:55.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f2275d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:55.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f2275d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:55.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f2275d9b640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:40:55.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.523+0000 7f2275d9b640 1 -- 172.21.15.37:0/4272711087 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f226000bd30 con 0x7f22781457f0 2024-03-29T11:40:55.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.524+0000 7f2275d9b640 1 -- 172.21.15.37:0/4272711087 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 819172074 0 0) 0x7f226000bd30 con 0x7f22781457f0 2024-03-29T11:40:55.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.524+0000 7f2275d9b640 1 -- 172.21.15.37:0/4272711087 >> v1:172.21.15.146:6789/0 conn(0x7f227813f650 legacy=0x7f22780b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:55.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.526+0000 7f2275d9b640 1 -- 172.21.15.37:0/4272711087 >> v1:172.21.15.37:6789/0 conn(0x7f22780a9d70 legacy=0x7f227814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:55.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.528+0000 7f2275d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:40:55.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.528+0000 7f2275d9b640 1 monclient: found mon.noname-b 2024-03-29T11:40:55.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.528+0000 7f2275d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:55.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.528+0000 7f2275d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:40:55.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.528+0000 7f2275d9b640 1 -- 172.21.15.37:0/4272711087 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2278146340 con 0x7f22781457f0 2024-03-29T11:40:55.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.529+0000 7f2275d9b640 10 monclient: _finish_auth 0 2024-03-29T11:40:55.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.529+0000 7f2275d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:40:55.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.529+0000 7f2275d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:55.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.529+0000 7f227d14d640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:40:55.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.529+0000 7f227d14d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:55.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.529+0000 7f2275d9b640 1 -- 172.21.15.37:0/4272711087 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2268001690 con 0x7f22781457f0 2024-03-29T11:40:55.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.529+0000 7f2275d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:55.533 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.529+0000 7f2275d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:55.533 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.529+0000 7f2275d9b640 10 monclient: dump: 2024-03-29T11:40:55.533 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:55.533 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:55.533 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:55.533 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:55.533 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:55.533 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:55.533 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:55.533 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:55.533 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:55.533 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:55.533 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.529+0000 7f227d14d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:55.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.530+0000 7f2275d9b640 1 -- 172.21.15.37:0/4272711087 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2268001920 con 0x7f22781457f0 2024-03-29T11:40:55.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.530+0000 7f2275d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:55.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.530+0000 7f227d14d640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:55.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.530+0000 7f227d14d640 4 set_mon_vals no callback set 2024-03-29T11:40:55.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.531+0000 7f2275d9b640 1 -- 172.21.15.37:0/4272711087 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2268001cb0 con 0x7f22781457f0 2024-03-29T11:40:55.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.531+0000 7f2275d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:55.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.531+0000 7f2275d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:55.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.531+0000 7f2275d9b640 10 monclient: dump: 2024-03-29T11:40:55.536 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:55.536 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:55.536 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:55.536 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:55.536 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:55.536 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:55.536 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:55.536 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:55.536 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:55.536 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:55.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.532+0000 7f227d14d640 10 monclient: shutdown 2024-03-29T11:40:55.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.532+0000 7f227d14d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:55.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.532+0000 7f227d14d640 1 -- 172.21.15.37:0/4272711087 >> v1:172.21.15.37:6790/0 conn(0x7f22781457f0 legacy=0x7f227813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:55.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.533+0000 7f227d14d640 1 -- 172.21.15.37:0/4272711087 shutdown_connections 2024-03-29T11:40:55.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.533+0000 7f227d14d640 1 -- 172.21.15.37:0/4272711087 >> 172.21.15.37:0/4272711087 conn(0x7f2278066a00 msgr2=0x7f2278066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:55.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.533+0000 7f227d14d640 1 -- 172.21.15.37:0/4272711087 shutdown_connections 2024-03-29T11:40:55.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.533+0000 7f227d14d640 1 -- 172.21.15.37:0/4272711087 wait complete. 2024-03-29T11:40:55.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.534+0000 7f227d14d640 10 monclient: build_initial_monmap 2024-03-29T11:40:55.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.534+0000 7f227d14d640 10 monclient: monmap: 2024-03-29T11:40:55.538 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:40:55.538 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:55.538 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:40:55.538 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:40:55.538 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:55.538 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:55.538 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:40:55.538 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:40:55.538 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:40:55.538 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:55.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.534+0000 7f227d14d640 1 librados: starting msgr at 2024-03-29T11:40:55.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.534+0000 7f227d14d640 1 librados: starting objecter 2024-03-29T11:40:55.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.535+0000 7f227d14d640 1 Processor -- start 2024-03-29T11:40:55.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.536+0000 7f227d14d640 1 -- start start 2024-03-29T11:40:55.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.536+0000 7f227d14d640 1 librados: setting wanted keys 2024-03-29T11:40:55.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.536+0000 7f227d14d640 1 librados: calling monclient init 2024-03-29T11:40:55.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.536+0000 7f227d14d640 10 monclient: init 2024-03-29T11:40:55.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.537+0000 7f227d14d640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:55.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.537+0000 7f227d14d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:55.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.537+0000 7f227d14d640 10 monclient(hunting): picked mon.noname-a con 0x7f22780a9d70 addr v1:172.21.15.37:6789/0 2024-03-29T11:40:55.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.537+0000 7f227d14d640 10 monclient(hunting): picked mon.noname-b con 0x7f227813f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:40:55.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.537+0000 7f227d14d640 10 monclient(hunting): picked mon.noname-c con 0x7f22781457f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:40:55.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.537+0000 7f227d14d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2278142150 con 0x7f22780a9d70 2024-03-29T11:40:55.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.537+0000 7f227d14d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f227806ee10 con 0x7f227813f650 2024-03-29T11:40:55.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.537+0000 7f227d14d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2278142d50 con 0x7f22781457f0 2024-03-29T11:40:55.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.537+0000 7f227d14d640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:55.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.537+0000 7f227d14d640 10 monclient(hunting): authenticate will time out at 1234.986362s 2024-03-29T11:40:55.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.540+0000 7f227759e640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f22781457f0 0x7f22781deab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:53210/0 (socket says 172.21.15.37:53210) 2024-03-29T11:40:55.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.540+0000 7f227759e640 1 -- 172.21.15.37:0/1061035316 learned_addr learned my addr 172.21.15.37:0/1061035316 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:40:55.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.541+0000 7f225f7fe640 10 client.?.objecter ms_handle_connect 0x7f22781457f0 2024-03-29T11:40:55.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.541+0000 7f225f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:55.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.541+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686567213 0 0) 0x7f2278142d50 con 0x7f22781457f0 2024-03-29T11:40:55.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.541+0000 7f225f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:55.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.541+0000 7f225f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:55.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.541+0000 7f225f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:55.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.541+0000 7f225f7fe640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:40:55.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.541+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2264004eb0 con 0x7f22781457f0 2024-03-29T11:40:55.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.541+0000 7f225f7fe640 10 client.?.objecter ms_handle_connect 0x7f22780a9d70 2024-03-29T11:40:55.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.541+0000 7f225f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:55.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 10 client.?.objecter ms_handle_connect 0x7f227813f650 2024-03-29T11:40:55.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:55.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745385252 0 0) 0x7f2278142150 con 0x7f22780a9d70 2024-03-29T11:40:55.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:55.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:55.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:55.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:40:55.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2278142d50 con 0x7f22780a9d70 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 893668972 0 0) 0x7f2264004eb0 con 0x7f22781457f0 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2278142150 con 0x7f22781457f0 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2270001be0 con 0x7f22781457f0 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 10 monclient(hunting): dump: 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:55.548 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:55.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.545+0000 7f225f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:55.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533882873 0 0) 0x7f227806ee10 con 0x7f227813f650 2024-03-29T11:40:55.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:55.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:55.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:55.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:40:55.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2264004eb0 con 0x7f227813f650 2024-03-29T11:40:55.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1495260745 0 0) 0x7f2278142150 con 0x7f22781457f0 2024-03-29T11:40:55.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 >> v1:172.21.15.146:6789/0 conn(0x7f227813f650 legacy=0x7f2278136e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:55.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 >> v1:172.21.15.37:6789/0 conn(0x7f22780a9d70 legacy=0x7f2278142640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:55.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:55.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 1 monclient: found mon.c 2024-03-29T11:40:55.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:55.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:55.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227806de90 con 0x7f22781457f0 2024-03-29T11:40:55.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:55.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:55.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f225f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:55.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f227d14d640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:40:55.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f227d14d640 10 monclient: _renew_subs 2024-03-29T11:40:55.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f227d14d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:55.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f227d14d640 1 -- 172.21.15.37:0/1061035316 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f227806e100 con 0x7f22781457f0 2024-03-29T11:40:55.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f227d14d640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:55.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f227d14d640 10 monclient: _renew_subs 2024-03-29T11:40:55.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f227d14d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:55.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f227d14d640 1 -- 172.21.15.37:0/1061035316 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2278137740 con 0x7f22781457f0 2024-03-29T11:40:55.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.547+0000 7f227d14d640 1 librados: init done 2024-03-29T11:40:55.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.548+0000 7f227d14d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:55.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.548+0000 7f227d14d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:55.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.548+0000 7f227d14d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:55.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.548+0000 7f227d14d640 1 -- 172.21.15.37:0/1061035316 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2278075f10 con 0x7f22781457f0 2024-03-29T11:40:55.555 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.549+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22700020f0 con 0x7f22781457f0 2024-03-29T11:40:55.555 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.549+0000 7f225f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:55.555 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.549+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f2270003b10 con 0x7f22781457f0 2024-03-29T11:40:55.555 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.549+0000 7f225f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:55.555 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.549+0000 7f225f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:55.555 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.549+0000 7f225f7fe640 10 monclient: dump: 2024-03-29T11:40:55.555 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:40:55.555 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:40:55.555 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:55.555 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:40:55.555 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:55.555 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:40:55.555 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:40:55.556 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:40:55.556 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:40:55.556 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:40:55.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.550+0000 7f227559a640 4 set_mon_vals no callback set 2024-03-29T11:40:55.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.551+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f2270003c70 con 0x7f22781457f0 2024-03-29T11:40:55.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.552+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f2270084bb0 con 0x7f22781457f0 2024-03-29T11:40:55.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.552+0000 7f225f7fe640 10 client.4475.objecter ms_dispatch 0x7f2278000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:55.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.552+0000 7f225f7fe640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:55.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.552+0000 7f225f7fe640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:55.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.552+0000 7f225f7fe640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:40:55.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.554+0000 7f2276d9d640 10 monclient: get_auth_request con 0x7f226406dc60 auth_method 0 2024-03-29T11:40:55.558 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.556+0000 7f225f7fe640 10 client.4475.objecter ms_handle_connect 0x7f226406dc60 2024-03-29T11:40:55.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.558+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2270050490 con 0x7f22781457f0 2024-03-29T11:40:55.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.559+0000 7f225f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:55.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.559+0000 7f225f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:55.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.896+0000 7f227d14d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:55.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.896+0000 7f227d14d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:55.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.896+0000 7f227d14d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:40:55.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.896+0000 7f227d14d640 1 -- 172.21.15.37:0/1061035316 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2278067180 con 0x7f22781457f0 2024-03-29T11:40:55.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.899+0000 7f225f7fe640 1 -- 172.21.15.37:0/1061035316 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f22700523c0 con 0x7f22781457f0 2024-03-29T11:40:55.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.899+0000 7f225f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:55.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.899+0000 7f225f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:55.901 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:40:55.901 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:55.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.902+0000 7f227d14d640 10 librados: watch_flush enter 2024-03-29T11:40:55.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.903+0000 7f227d14d640 10 librados: watch_flush exit 2024-03-29T11:40:55.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.903+0000 7f227d14d640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:40:55.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.903+0000 7f227d14d640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:40:55.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.904+0000 7f227d14d640 1 -- 172.21.15.37:0/1061035316 >> v1:172.21.15.146:6816/2899149139 conn(0x7f226406dc60 legacy=0x7f2264070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:55.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.904+0000 7f227d14d640 10 monclient: shutdown 2024-03-29T11:40:55.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.904+0000 7f227d14d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:55.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.904+0000 7f227d14d640 1 -- 172.21.15.37:0/1061035316 >> v1:172.21.15.37:6790/0 conn(0x7f22781457f0 legacy=0x7f22781deab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:55.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.904+0000 7f227d14d640 1 -- 172.21.15.37:0/1061035316 shutdown_connections 2024-03-29T11:40:55.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.904+0000 7f227d14d640 1 -- 172.21.15.37:0/1061035316 >> 172.21.15.37:0/1061035316 conn(0x7f2278066a00 msgr2=0x7f227813a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:55.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.904+0000 7f227d14d640 1 -- 172.21.15.37:0/1061035316 shutdown_connections 2024-03-29T11:40:55.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.904+0000 7f227d14d640 1 -- 172.21.15.37:0/1061035316 wait complete. 2024-03-29T11:40:55.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:40:55.904+0000 7f227d14d640 1 librados: shutdown 2024-03-29T11:40:55.924 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:02.730 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:02.875 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.873+0000 7f6f768f2640 10 monclient: get_monmap_and_config 2024-03-29T11:41:02.875 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.873+0000 7f6f768f2640 10 monclient: build_initial_monmap 2024-03-29T11:41:02.875 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.873+0000 7f6f768f2640 10 monclient: monmap: 2024-03-29T11:41:02.875 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:02.875 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:02.875 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:41:02.874061+0000 2024-03-29T11:41:02.876 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:41:02.874061+0000 2024-03-29T11:41:02.876 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:02.876 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:02.876 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:02.876 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:41:02.876 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:41:02.876 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:02.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.874+0000 7f6f768f2640 1 Processor -- start 2024-03-29T11:41:02.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.874+0000 7f6f768f2640 1 -- start start 2024-03-29T11:41:02.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.874+0000 7f6f768f2640 10 monclient: init 2024-03-29T11:41:02.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.876+0000 7f6f768f2640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:02.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.876+0000 7f6f768f2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:02.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.876+0000 7f6f768f2640 10 monclient(hunting): picked mon.noname-b con 0x7f6f700b2800 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:02.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.876+0000 7f6f768f2640 10 monclient(hunting): picked mon.noname-c con 0x7f6f700b3220 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:02.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.876+0000 7f6f768f2640 10 monclient(hunting): picked mon.noname-a con 0x7f6f701439c0 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:02.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.876+0000 7f6f768f2640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f700b3840 con 0x7f6f701439c0 2024-03-29T11:41:02.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.876+0000 7f6f768f2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7006f230 con 0x7f6f700b3220 2024-03-29T11:41:02.879 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.876+0000 7f6f768f2640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7006ef50 con 0x7f6f700b2800 2024-03-29T11:41:02.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.876+0000 7f6f768f2640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:02.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.876+0000 7f6f768f2640 10 monclient(hunting): authenticate will time out at 1242.325251s 2024-03-29T11:41:02.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.878+0000 7f6f758f0640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f6f700b2800 0x7f6f701432b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:54334/0 (socket says 172.21.15.37:54334) 2024-03-29T11:41:02.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.878+0000 7f6f758f0640 1 -- 172.21.15.37:0/1832915795 learned_addr learned my addr 172.21.15.37:0/1832915795 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:02.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.880+0000 7f6f748ee640 1 -- 172.21.15.37:0/1832915795 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383457956 0 0) 0x7f6f7006ef50 con 0x7f6f700b2800 2024-03-29T11:41:02.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.880+0000 7f6f748ee640 10 monclient(hunting): _negotiate 2024-03-29T11:41:02.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.880+0000 7f6f748ee640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:02.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.880+0000 7f6f748ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:02.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.880+0000 7f6f748ee640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:41:02.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.880+0000 7f6f748ee640 1 -- 172.21.15.37:0/1832915795 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f58003520 con 0x7f6f700b2800 2024-03-29T11:41:02.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.881+0000 7f6f748ee640 1 -- 172.21.15.37:0/1832915795 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243178428 0 0) 0x7f6f7006f230 con 0x7f6f700b3220 2024-03-29T11:41:02.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.881+0000 7f6f748ee640 10 monclient(hunting): _negotiate 2024-03-29T11:41:02.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.881+0000 7f6f748ee640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:02.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.881+0000 7f6f748ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:02.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.881+0000 7f6f748ee640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:41:02.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.881+0000 7f6f748ee640 1 -- 172.21.15.37:0/1832915795 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f7006ef50 con 0x7f6f700b3220 2024-03-29T11:41:02.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.882+0000 7f6f748ee640 1 -- 172.21.15.37:0/1832915795 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4026063528 0 0) 0x7f6f7006ef50 con 0x7f6f700b3220 2024-03-29T11:41:02.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.882+0000 7f6f748ee640 1 -- 172.21.15.37:0/1832915795 >> v1:172.21.15.37:6790/0 conn(0x7f6f700b2800 legacy=0x7f6f701432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:02.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 1 -- 172.21.15.37:0/1832915795 >> v1:172.21.15.37:6789/0 conn(0x7f6f701439c0 legacy=0x7f6f70151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:02.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 10 monclient: _finish_hunting 0 2024-03-29T11:41:02.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 1 monclient: found mon.noname-c 2024-03-29T11:41:02.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:02.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:41:02.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 1 -- 172.21.15.37:0/1832915795 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f70071d30 con 0x7f6f700b3220 2024-03-29T11:41:02.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 10 monclient: _finish_auth 0 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 10 monclient: _check_auth_tickets 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 1 -- 172.21.15.37:0/1832915795 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6f68001670 con 0x7f6f700b3220 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 10 monclient: dump: 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:02.888 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:02.889 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:02.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 1 -- 172.21.15.37:0/1832915795 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f680017d0 con 0x7f6f700b3220 2024-03-29T11:41:02.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:02.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 1 -- 172.21.15.37:0/1832915795 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6f68001be0 con 0x7f6f700b3220 2024-03-29T11:41:02.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:02.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:02.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.883+0000 7f6f748ee640 10 monclient: dump: 2024-03-29T11:41:02.889 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:02.889 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:02.889 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:02.889 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:02.889 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:02.889 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:02.889 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:02.890 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:02.890 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:02.890 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:02.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.885+0000 7f6f768f2640 5 monclient: authenticate success, global_id 4399 2024-03-29T11:41:02.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.885+0000 7f6f768f2640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:02.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.885+0000 7f6f768f2640 4 set_mon_vals no callback set 2024-03-29T11:41:02.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.886+0000 7f6f768f2640 10 monclient: shutdown 2024-03-29T11:41:02.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.886+0000 7f6f768f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:02.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.886+0000 7f6f768f2640 1 -- 172.21.15.37:0/1832915795 >> v1:172.21.15.146:6789/0 conn(0x7f6f700b3220 legacy=0x7f6f70146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:02.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.887+0000 7f6f768f2640 1 -- 172.21.15.37:0/1832915795 shutdown_connections 2024-03-29T11:41:02.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.887+0000 7f6f768f2640 1 -- 172.21.15.37:0/1832915795 >> 172.21.15.37:0/1832915795 conn(0x7f6f70066a00 msgr2=0x7f6f70066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:02.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.887+0000 7f6f768f2640 1 -- 172.21.15.37:0/1832915795 shutdown_connections 2024-03-29T11:41:02.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.887+0000 7f6f768f2640 1 -- 172.21.15.37:0/1832915795 wait complete. 2024-03-29T11:41:02.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.889+0000 7f6f768f2640 10 monclient: build_initial_monmap 2024-03-29T11:41:02.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.889+0000 7f6f768f2640 10 monclient: monmap: 2024-03-29T11:41:02.892 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:02.892 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:02.892 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:41:02.892 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:41:02.892 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:02.892 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:02.892 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:02.892 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:41:02.892 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:41:02.892 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:02.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.889+0000 7f6f768f2640 1 librados: starting msgr at 2024-03-29T11:41:02.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.889+0000 7f6f768f2640 1 librados: starting objecter 2024-03-29T11:41:02.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.890+0000 7f6f768f2640 1 Processor -- start 2024-03-29T11:41:02.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.890+0000 7f6f768f2640 1 -- start start 2024-03-29T11:41:02.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.890+0000 7f6f768f2640 1 librados: setting wanted keys 2024-03-29T11:41:02.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.890+0000 7f6f768f2640 1 librados: calling monclient init 2024-03-29T11:41:02.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.890+0000 7f6f768f2640 10 monclient: init 2024-03-29T11:41:02.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.891+0000 7f6f768f2640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:02.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.891+0000 7f6f768f2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:02.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.892+0000 7f6f768f2640 10 monclient(hunting): picked mon.noname-c con 0x7f6f700b2800 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:02.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.893+0000 7f6f758f0640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f6f700b2800 0x7f6f70147660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:54334/0 (socket says 172.21.15.37:54334) 2024-03-29T11:41:02.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f768f2640 10 monclient(hunting): picked mon.noname-a con 0x7f6f700b3220 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:02.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f768f2640 10 monclient(hunting): picked mon.noname-b con 0x7f6f701439c0 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:02.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f768f2640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f70148b30 con 0x7f6f700b3220 2024-03-29T11:41:02.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f768f2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f701db230 con 0x7f6f701439c0 2024-03-29T11:41:02.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f768f2640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f701dbe60 con 0x7f6f700b2800 2024-03-29T11:41:02.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f768f2640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:02.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f768f2640 10 monclient(hunting): authenticate will time out at 1242.344182s 2024-03-29T11:41:02.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f758f0640 1 -- 172.21.15.37:0/604059785 learned_addr learned my addr 172.21.15.37:0/604059785 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:02.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f6e7fc640 10 client.?.objecter ms_handle_connect 0x7f6f700b2800 2024-03-29T11:41:02.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:02.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681786614 0 0) 0x7f6f701dbe60 con 0x7f6f700b2800 2024-03-29T11:41:02.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:02.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:02.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:02.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f6e7fc640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:41:02.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.895+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f5c004eb0 con 0x7f6f700b2800 2024-03-29T11:41:02.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 client.?.objecter ms_handle_connect 0x7f6f700b3220 2024-03-29T11:41:02.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:02.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 client.?.objecter ms_handle_connect 0x7f6f701439c0 2024-03-29T11:41:02.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:02.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295294216 0 0) 0x7f6f70148b30 con 0x7f6f700b3220 2024-03-29T11:41:02.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:02.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:02.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:02.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:41:02.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f701dbe60 con 0x7f6f700b3220 2024-03-29T11:41:02.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 212935254 0 0) 0x7f6f5c004eb0 con 0x7f6f700b2800 2024-03-29T11:41:02.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f70148b30 con 0x7f6f700b2800 2024-03-29T11:41:02.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6f60001c50 con 0x7f6f700b2800 2024-03-29T11:41:02.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:02.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:02.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 monclient(hunting): dump: 2024-03-29T11:41:02.903 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:02.903 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:02.903 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:02.903 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:02.903 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:02.903 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:02.903 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:02.903 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:02.903 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:02.903 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:02.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:02.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205366666 0 0) 0x7f6f701db230 con 0x7f6f701439c0 2024-03-29T11:41:02.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:02.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:02.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:02.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:41:02.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.899+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f5c004eb0 con 0x7f6f701439c0 2024-03-29T11:41:02.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3326315443 0 0) 0x7f6f5c004eb0 con 0x7f6f701439c0 2024-03-29T11:41:02.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f701db230 con 0x7f6f701439c0 2024-03-29T11:41:02.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3158527024 0 0) 0x7f6f70148b30 con 0x7f6f700b2800 2024-03-29T11:41:02.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 >> v1:172.21.15.146:6789/0 conn(0x7f6f701439c0 legacy=0x7f6f701e5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:02.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 >> v1:172.21.15.37:6789/0 conn(0x7f6f700b3220 legacy=0x7f6f701db750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:02.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:02.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 1 monclient: found mon.c 2024-03-29T11:41:02.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:02.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:02.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f701dbff0 con 0x7f6f700b2800 2024-03-29T11:41:02.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:02.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:02.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:02.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f60002080 con 0x7f6f700b2800 2024-03-29T11:41:02.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:02.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6f60004d00 con 0x7f6f700b2800 2024-03-29T11:41:02.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:02.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:02.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.900+0000 7f6f6e7fc640 10 monclient: dump: 2024-03-29T11:41:02.907 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:02.907 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:02.907 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:02.907 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:02.907 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:02.907 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:02.907 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:02.907 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:02.907 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:02.908 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:02.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.901+0000 7f6f768f2640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:41:02.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.901+0000 7f6f768f2640 10 monclient: _renew_subs 2024-03-29T11:41:02.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.901+0000 7f6f768f2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:02.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.901+0000 7f6f768f2640 1 -- 172.21.15.37:0/604059785 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f70149230 con 0x7f6f700b2800 2024-03-29T11:41:02.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.901+0000 7f6f768f2640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:02.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.903+0000 7f6f6f7fe640 4 set_mon_vals no callback set 2024-03-29T11:41:02.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.904+0000 7f6f768f2640 10 monclient: _renew_subs 2024-03-29T11:41:02.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.904+0000 7f6f768f2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:02.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.904+0000 7f6f768f2640 1 -- 172.21.15.37:0/604059785 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7013ee50 con 0x7f6f700b2800 2024-03-29T11:41:02.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.904+0000 7f6f768f2640 1 librados: init done 2024-03-29T11:41:02.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.906+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f6f60004e60 con 0x7f6f700b2800 2024-03-29T11:41:02.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.906+0000 7f6f768f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:02.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.906+0000 7f6f768f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:02.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.906+0000 7f6f768f2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:02.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.906+0000 7f6f768f2640 1 -- 172.21.15.37:0/604059785 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f70070e20 con 0x7f6f700b2800 2024-03-29T11:41:02.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.907+0000 7f6f750ef640 10 monclient: get_auth_request con 0x7f6f5c06dd90 auth_method 0 2024-03-29T11:41:02.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.907+0000 7f6f6e7fc640 10 client.4481.objecter ms_handle_connect 0x7f6f5c06dd90 2024-03-29T11:41:02.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.910+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f6f60054b50 con 0x7f6f700b2800 2024-03-29T11:41:02.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.910+0000 7f6f6e7fc640 10 client.4481.objecter ms_dispatch 0x7f6f70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:02.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.910+0000 7f6f6e7fc640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:02.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.910+0000 7f6f6e7fc640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:02.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.910+0000 7f6f6e7fc640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T11:41:02.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.916+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6f60050df0 con 0x7f6f700b2800 2024-03-29T11:41:02.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.916+0000 7f6f6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:02.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:02.916+0000 7f6f6e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:03.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.238+0000 7f6f768f2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:03.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.238+0000 7f6f768f2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:03.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.238+0000 7f6f768f2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:03.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.238+0000 7f6f768f2640 1 -- 172.21.15.37:0/604059785 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f701db3c0 con 0x7f6f700b2800 2024-03-29T11:41:03.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.240+0000 7f6f6e7fc640 1 -- 172.21.15.37:0/604059785 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f6f60050f90 con 0x7f6f700b2800 2024-03-29T11:41:03.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.240+0000 7f6f6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:03.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.240+0000 7f6f6e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:03.246 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:41:03.246 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:03.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.244+0000 7f6f768f2640 10 librados: watch_flush enter 2024-03-29T11:41:03.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.244+0000 7f6f768f2640 10 librados: watch_flush exit 2024-03-29T11:41:03.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.244+0000 7f6f768f2640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T11:41:03.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.244+0000 7f6f768f2640 10 client.4481.objecter successfully canceled tick 2024-03-29T11:41:03.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.244+0000 7f6f768f2640 1 -- 172.21.15.37:0/604059785 >> v1:172.21.15.146:6816/2899149139 conn(0x7f6f5c06dd90 legacy=0x7f6f5c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.244+0000 7f6f768f2640 10 monclient: shutdown 2024-03-29T11:41:03.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.244+0000 7f6f768f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:03.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.244+0000 7f6f768f2640 1 -- 172.21.15.37:0/604059785 >> v1:172.21.15.37:6790/0 conn(0x7f6f700b2800 legacy=0x7f6f70147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.244+0000 7f6f768f2640 1 -- 172.21.15.37:0/604059785 shutdown_connections 2024-03-29T11:41:03.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.244+0000 7f6f768f2640 1 -- 172.21.15.37:0/604059785 >> 172.21.15.37:0/604059785 conn(0x7f6f70066a00 msgr2=0x7f6f700b7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:03.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.245+0000 7f6f768f2640 1 -- 172.21.15.37:0/604059785 shutdown_connections 2024-03-29T11:41:03.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.245+0000 7f6f768f2640 1 -- 172.21.15.37:0/604059785 wait complete. 2024-03-29T11:41:03.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:03.245+0000 7f6f768f2640 1 librados: shutdown 2024-03-29T11:41:03.264 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:10.071 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:10.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.223+0000 7f17c456c640 10 monclient: get_monmap_and_config 2024-03-29T11:41:10.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.223+0000 7f17c456c640 10 monclient: build_initial_monmap 2024-03-29T11:41:10.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.223+0000 7f17c456c640 10 monclient: monmap: 2024-03-29T11:41:10.226 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:10.226 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:10.227 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:41:10.224747+0000 2024-03-29T11:41:10.227 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:41:10.224747+0000 2024-03-29T11:41:10.227 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:10.227 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:10.227 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:10.227 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:41:10.227 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:41:10.227 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:10.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.224+0000 7f17c456c640 1 Processor -- start 2024-03-29T11:41:10.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.224+0000 7f17c456c640 1 -- start start 2024-03-29T11:41:10.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.224+0000 7f17c456c640 10 monclient: init 2024-03-29T11:41:10.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.225+0000 7f17c456c640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:10.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.225+0000 7f17c456c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:10.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.225+0000 7f17c456c640 10 monclient(hunting): picked mon.noname-a con 0x7f17bc1457f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:10.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.225+0000 7f17c456c640 10 monclient(hunting): picked mon.noname-c con 0x7f17bc13f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:10.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.225+0000 7f17c456c640 10 monclient(hunting): picked mon.noname-b con 0x7f17bc0a9d70 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:10.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.225+0000 7f17c456c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc0aaef0 con 0x7f17bc1457f0 2024-03-29T11:41:10.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.225+0000 7f17c456c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc145bd0 con 0x7f17bc13f650 2024-03-29T11:41:10.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.225+0000 7f17c456c640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc145eb0 con 0x7f17bc0a9d70 2024-03-29T11:41:10.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.225+0000 7f17c456c640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:10.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.225+0000 7f17c456c640 10 monclient(hunting): authenticate will time out at 1249.674880s 2024-03-29T11:41:10.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.228+0000 7f17c2b09640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f17bc0a9d70 0x7f17bc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:57955/0 (socket says 172.21.15.37:57955) 2024-03-29T11:41:10.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.228+0000 7f17c2b09640 1 -- 172.21.15.37:0/3619795431 learned_addr learned my addr 172.21.15.37:0/3619795431 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:10.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.230+0000 7f17c1306640 1 -- 172.21.15.37:0/3619795431 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317977435 0 0) 0x7f17bc145eb0 con 0x7f17bc0a9d70 2024-03-29T11:41:10.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.230+0000 7f17c1306640 10 monclient(hunting): _negotiate 2024-03-29T11:41:10.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.230+0000 7f17c1306640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:10.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.230+0000 7f17c1306640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:10.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.230+0000 7f17c1306640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:41:10.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.230+0000 7f17c1306640 1 -- 172.21.15.37:0/3619795431 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a8003520 con 0x7f17bc0a9d70 2024-03-29T11:41:10.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.231+0000 7f17c1306640 1 -- 172.21.15.37:0/3619795431 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2182463592 0 0) 0x7f17a8003520 con 0x7f17bc0a9d70 2024-03-29T11:41:10.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.231+0000 7f17c1306640 1 -- 172.21.15.37:0/3619795431 >> v1:172.21.15.146:6789/0 conn(0x7f17bc13f650 legacy=0x7f17bc0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:10.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.231+0000 7f17c1306640 1 -- 172.21.15.37:0/3619795431 >> v1:172.21.15.37:6789/0 conn(0x7f17bc1457f0 legacy=0x7f17bc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.231+0000 7f17c1306640 10 monclient: _finish_hunting 0 2024-03-29T11:41:10.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.231+0000 7f17c1306640 1 monclient: found mon.noname-b 2024-03-29T11:41:10.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.231+0000 7f17c1306640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:10.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.231+0000 7f17c1306640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:41:10.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.232+0000 7f17c1306640 1 -- 172.21.15.37:0/3619795431 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc146340 con 0x7f17bc0a9d70 2024-03-29T11:41:10.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.232+0000 7f17c1306640 10 monclient: _finish_auth 0 2024-03-29T11:41:10.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.232+0000 7f17c1306640 10 monclient: _check_auth_tickets 2024-03-29T11:41:10.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.232+0000 7f17c1306640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:10.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.232+0000 7f17c456c640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:41:10.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.232+0000 7f17c456c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:10.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.232+0000 7f17c1306640 1 -- 172.21.15.37:0/3619795431 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f17b8001670 con 0x7f17bc0a9d70 2024-03-29T11:41:10.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.232+0000 7f17c1306640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:10.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.232+0000 7f17c1306640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:10.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.232+0000 7f17c1306640 10 monclient: dump: 2024-03-29T11:41:10.236 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:10.236 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:10.236 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:10.236 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:10.236 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:10.236 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:10.236 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:10.236 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:10.236 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:10.236 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:10.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.232+0000 7f17c456c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:10.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.233+0000 7f17c1306640 1 -- 172.21.15.37:0/3619795431 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17b8001900 con 0x7f17bc0a9d70 2024-03-29T11:41:10.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.233+0000 7f17c1306640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:10.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.233+0000 7f17c456c640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:10.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.233+0000 7f17c456c640 4 set_mon_vals no callback set 2024-03-29T11:41:10.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.234+0000 7f17c456c640 10 monclient: shutdown 2024-03-29T11:41:10.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.234+0000 7f17c456c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:10.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.234+0000 7f17c456c640 1 -- 172.21.15.37:0/3619795431 >> v1:172.21.15.37:6790/0 conn(0x7f17bc0a9d70 legacy=0x7f17bc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.236+0000 7f17c456c640 1 -- 172.21.15.37:0/3619795431 shutdown_connections 2024-03-29T11:41:10.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.236+0000 7f17c456c640 1 -- 172.21.15.37:0/3619795431 >> 172.21.15.37:0/3619795431 conn(0x7f17bc066a00 msgr2=0x7f17bc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:10.240 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.238+0000 7f17c456c640 1 -- 172.21.15.37:0/3619795431 shutdown_connections 2024-03-29T11:41:10.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.241+0000 7f17c456c640 1 -- 172.21.15.37:0/3619795431 wait complete. 2024-03-29T11:41:10.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.242+0000 7f17c456c640 10 monclient: build_initial_monmap 2024-03-29T11:41:10.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.242+0000 7f17c456c640 10 monclient: monmap: 2024-03-29T11:41:10.245 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:10.245 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:10.245 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:41:10.245 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:41:10.245 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:10.245 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:10.245 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:10.245 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:41:10.245 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:41:10.245 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:10.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.243+0000 7f17c456c640 1 librados: starting msgr at 2024-03-29T11:41:10.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.243+0000 7f17c456c640 1 librados: starting objecter 2024-03-29T11:41:10.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.243+0000 7f17c456c640 1 Processor -- start 2024-03-29T11:41:10.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.243+0000 7f17c456c640 1 -- start start 2024-03-29T11:41:10.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.243+0000 7f17c456c640 1 librados: setting wanted keys 2024-03-29T11:41:10.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.243+0000 7f17c456c640 1 librados: calling monclient init 2024-03-29T11:41:10.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.243+0000 7f17c456c640 10 monclient: init 2024-03-29T11:41:10.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.244+0000 7f17c456c640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:10.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.244+0000 7f17c456c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:10.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.244+0000 7f17c456c640 10 monclient(hunting): picked mon.noname-a con 0x7f17bc0a9d70 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:10.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.244+0000 7f17c456c640 10 monclient(hunting): picked mon.noname-c con 0x7f17bc13f650 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:10.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.244+0000 7f17c456c640 10 monclient(hunting): picked mon.noname-b con 0x7f17bc1457f0 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:10.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.244+0000 7f17c456c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc13ba20 con 0x7f17bc0a9d70 2024-03-29T11:41:10.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.244+0000 7f17c456c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc06de90 con 0x7f17bc1457f0 2024-03-29T11:41:10.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.244+0000 7f17c456c640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc071f90 con 0x7f17bc13f650 2024-03-29T11:41:10.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.244+0000 7f17c456c640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:10.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.244+0000 7f17c456c640 10 monclient(hunting): authenticate will time out at 1249.693222s 2024-03-29T11:41:10.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.247+0000 7f17c1b07640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f17bc13f650 0x7f17bc141410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:58107/0 (socket says 172.21.15.37:58107) 2024-03-29T11:41:10.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.247+0000 7f17c1b07640 1 -- 172.21.15.37:0/1674079149 learned_addr learned my addr 172.21.15.37:0/1674079149 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:10.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 10 client.?.objecter ms_handle_connect 0x7f17bc13f650 2024-03-29T11:41:10.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:10.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 10 client.?.objecter ms_handle_connect 0x7f17bc0a9d70 2024-03-29T11:41:10.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:10.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148344044 0 0) 0x7f17bc071f90 con 0x7f17bc13f650 2024-03-29T11:41:10.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:10.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:10.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:10.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:41:10.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a0004eb0 con 0x7f17bc13f650 2024-03-29T11:41:10.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546957671 0 0) 0x7f17bc13ba20 con 0x7f17bc0a9d70 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.249+0000 7f17b2ffd640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17bc071f90 con 0x7f17bc0a9d70 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2725468537 0 0) 0x7f17a0004eb0 con 0x7f17bc13f650 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17bc13ba20 con 0x7f17bc13f650 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f17ac001870 con 0x7f17bc13f650 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:10.256 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:10.257 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:10.257 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:10.257 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:10.257 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:10.257 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:10.257 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:10.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:10.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 788393118 0 0) 0x7f17bc071f90 con 0x7f17bc0a9d70 2024-03-29T11:41:10.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17a0004eb0 con 0x7f17bc0a9d70 2024-03-29T11:41:10.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4175837140 0 0) 0x7f17bc13ba20 con 0x7f17bc13f650 2024-03-29T11:41:10.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 >> v1:172.21.15.146:6789/0 conn(0x7f17bc1457f0 legacy=0x7f17bc141b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 >> v1:172.21.15.37:6789/0 conn(0x7f17bc0a9d70 legacy=0x7f17bc137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:10.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 1 monclient: found mon.c 2024-03-29T11:41:10.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:10.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:10.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc138090 con 0x7f17bc13f650 2024-03-29T11:41:10.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:10.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:10.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:10.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17c456c640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:41:10.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17c456c640 10 monclient: _renew_subs 2024-03-29T11:41:10.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17c456c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:10.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.250+0000 7f17c456c640 1 -- 172.21.15.37:0/1674079149 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17bc1dece0 con 0x7f17bc13f650 2024-03-29T11:41:10.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.251+0000 7f17c456c640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:10.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.251+0000 7f17c456c640 10 monclient: _renew_subs 2024-03-29T11:41:10.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.251+0000 7f17c456c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.251+0000 7f17c456c640 1 -- 172.21.15.37:0/1674079149 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17bc1df210 con 0x7f17bc13f650 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.251+0000 7f17c456c640 1 librados: init done 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.251+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17ac0019d0 con 0x7f17bc13f650 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.251+0000 7f17b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.251+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f17ac004aa0 con 0x7f17bc13f650 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.251+0000 7f17b3fff640 4 set_mon_vals no callback set 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.251+0000 7f17b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.251+0000 7f17b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.251+0000 7f17b2ffd640 10 monclient: dump: 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:10.259 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:10.260 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:10.260 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:10.260 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:10.260 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:10.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.252+0000 7f17c456c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:10.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.252+0000 7f17c456c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:10.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.252+0000 7f17c456c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:10.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.252+0000 7f17c456c640 1 -- 172.21.15.37:0/1674079149 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a4005080 con 0x7f17bc13f650 2024-03-29T11:41:10.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.256+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f17ac01aa70 con 0x7f17bc13f650 2024-03-29T11:41:10.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.256+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f17ac084a60 con 0x7f17bc13f650 2024-03-29T11:41:10.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.256+0000 7f17b2ffd640 10 client.4487.objecter ms_dispatch 0x7f17bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:10.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.256+0000 7f17b2ffd640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:10.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.256+0000 7f17b2ffd640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:10.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.256+0000 7f17b2ffd640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T11:41:10.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.258+0000 7f17c2308640 10 monclient: get_auth_request con 0x7f17a006de80 auth_method 0 2024-03-29T11:41:10.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.260+0000 7f17b2ffd640 10 client.4487.objecter ms_handle_connect 0x7f17a006de80 2024-03-29T11:41:10.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.266+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17ac050170 con 0x7f17bc13f650 2024-03-29T11:41:10.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.267+0000 7f17b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:10.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.267+0000 7f17b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:10.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.608+0000 7f17c456c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:10.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.608+0000 7f17c456c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:10.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.608+0000 7f17c456c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:10.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.608+0000 7f17c456c640 1 -- 172.21.15.37:0/1674079149 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17a4005330 con 0x7f17bc13f650 2024-03-29T11:41:10.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.611+0000 7f17b2ffd640 1 -- 172.21.15.37:0/1674079149 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f17ac052270 con 0x7f17bc13f650 2024-03-29T11:41:10.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.611+0000 7f17b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:10.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.611+0000 7f17b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:10.613 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:41:10.613 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:10.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.613+0000 7f17c456c640 10 librados: watch_flush enter 2024-03-29T11:41:10.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.613+0000 7f17c456c640 10 librados: watch_flush exit 2024-03-29T11:41:10.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.613+0000 7f17c456c640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T11:41:10.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.613+0000 7f17c456c640 10 client.4487.objecter successfully canceled tick 2024-03-29T11:41:10.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.614+0000 7f17c456c640 1 -- 172.21.15.37:0/1674079149 >> v1:172.21.15.146:6816/2899149139 conn(0x7f17a006de80 legacy=0x7f17a0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.614+0000 7f17c456c640 10 monclient: shutdown 2024-03-29T11:41:10.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.614+0000 7f17c456c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:10.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.614+0000 7f17c456c640 1 -- 172.21.15.37:0/1674079149 >> v1:172.21.15.37:6790/0 conn(0x7f17bc13f650 legacy=0x7f17bc141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.617+0000 7f17c456c640 1 -- 172.21.15.37:0/1674079149 shutdown_connections 2024-03-29T11:41:10.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.617+0000 7f17c456c640 1 -- 172.21.15.37:0/1674079149 >> 172.21.15.37:0/1674079149 conn(0x7f17bc066a00 msgr2=0x7f17bc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:10.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.617+0000 7f17c456c640 1 -- 172.21.15.37:0/1674079149 shutdown_connections 2024-03-29T11:41:10.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.617+0000 7f17c456c640 1 -- 172.21.15.37:0/1674079149 wait complete. 2024-03-29T11:41:10.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:10.617+0000 7f17c456c640 1 librados: shutdown 2024-03-29T11:41:10.636 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:17.444 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:17.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.604+0000 7f6ad7d0e640 10 monclient: get_monmap_and_config 2024-03-29T11:41:17.608 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.604+0000 7f6ad7d0e640 10 monclient: build_initial_monmap 2024-03-29T11:41:17.608 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.604+0000 7f6ad7d0e640 10 monclient: monmap: 2024-03-29T11:41:17.608 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:17.608 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:17.608 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:41:17.605641+0000 2024-03-29T11:41:17.608 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:41:17.605641+0000 2024-03-29T11:41:17.608 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:17.608 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:17.608 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:17.608 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:41:17.608 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:41:17.608 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:17.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.605+0000 7f6ad7d0e640 1 Processor -- start 2024-03-29T11:41:17.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.606+0000 7f6ad7d0e640 1 -- start start 2024-03-29T11:41:17.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.606+0000 7f6ad7d0e640 10 monclient: init 2024-03-29T11:41:17.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.606+0000 7f6ad7d0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:17.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.607+0000 7f6ad7d0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:17.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.607+0000 7f6ad7d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f6ad00b4100 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:17.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.607+0000 7f6ad7d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f6ad013ee90 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:17.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.607+0000 7f6ad7d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f6ad00af900 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:17.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.607+0000 7f6ad7d0e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad00afd10 con 0x7f6ad00b4100 2024-03-29T11:41:17.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.607+0000 7f6ad7d0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad00af6f0 con 0x7f6ad013ee90 2024-03-29T11:41:17.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.607+0000 7f6ad7d0e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad00aac80 con 0x7f6ad00af900 2024-03-29T11:41:17.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.607+0000 7f6ad7d0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:17.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.607+0000 7f6ad7d0e640 10 monclient(hunting): authenticate will time out at 1257.056048s 2024-03-29T11:41:17.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.610+0000 7f6ad62ab640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f6ad00af900 0x7f6ad014be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:41507/0 (socket says 172.21.15.37:41507) 2024-03-29T11:41:17.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.610+0000 7f6ad62ab640 1 -- 172.21.15.37:0/202443547 learned_addr learned my addr 172.21.15.37:0/202443547 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:17.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.611+0000 7f6ad4aa8640 1 -- 172.21.15.37:0/202443547 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087532382 0 0) 0x7f6ad00aac80 con 0x7f6ad00af900 2024-03-29T11:41:17.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.611+0000 7f6ad4aa8640 10 monclient(hunting): _negotiate 2024-03-29T11:41:17.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.611+0000 7f6ad4aa8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:17.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.611+0000 7f6ad4aa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:17.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.611+0000 7f6ad4aa8640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:41:17.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.611+0000 7f6ad4aa8640 1 -- 172.21.15.37:0/202443547 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6abc003520 con 0x7f6ad00af900 2024-03-29T11:41:17.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.612+0000 7f6ad4aa8640 1 -- 172.21.15.37:0/202443547 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4182872816 0 0) 0x7f6abc003520 con 0x7f6ad00af900 2024-03-29T11:41:17.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.612+0000 7f6ad4aa8640 1 -- 172.21.15.37:0/202443547 >> v1:172.21.15.146:6789/0 conn(0x7f6ad013ee90 legacy=0x7f6ad013b5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:41:17.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.612+0000 7f6ad4aa8640 1 -- 172.21.15.37:0/202443547 >> v1:172.21.15.37:6789/0 conn(0x7f6ad00b4100 legacy=0x7f6ad00adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:17.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 10 monclient: _finish_hunting 0 2024-03-29T11:41:17.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 1 monclient: found mon.noname-b 2024-03-29T11:41:17.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:17.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:41:17.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 1 -- 172.21.15.37:0/202443547 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad00aaf60 con 0x7f6ad00af900 2024-03-29T11:41:17.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 10 monclient: _finish_auth 0 2024-03-29T11:41:17.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 10 monclient: _check_auth_tickets 2024-03-29T11:41:17.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:17.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 1 -- 172.21.15.37:0/202443547 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6ac80019f0 con 0x7f6ad00af900 2024-03-29T11:41:17.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:17.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:17.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 10 monclient: dump: 2024-03-29T11:41:17.616 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:17.616 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:17.616 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:17.616 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:17.616 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:17.616 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:17.616 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:17.616 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:17.616 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:17.616 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:17.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad7d0e640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:41:17.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad7d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:17.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 1 -- 172.21.15.37:0/202443547 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ac8001c50 con 0x7f6ad00af900 2024-03-29T11:41:17.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:17.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad7d0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:17.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad7d0e640 4 set_mon_vals no callback set 2024-03-29T11:41:17.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 1 -- 172.21.15.37:0/202443547 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6ac80020a0 con 0x7f6ad00af900 2024-03-29T11:41:17.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.613+0000 7f6ad4aa8640 10 monclient: dump: 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.614+0000 7f6ad7d0e640 10 monclient: shutdown 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.614+0000 7f6ad7d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:17.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.614+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/202443547 >> v1:172.21.15.37:6790/0 conn(0x7f6ad00af900 legacy=0x7f6ad014be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.614+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/202443547 shutdown_connections 2024-03-29T11:41:17.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.614+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/202443547 >> 172.21.15.37:0/202443547 conn(0x7f6ad00b35e0 msgr2=0x7f6ad00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:17.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.615+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/202443547 shutdown_connections 2024-03-29T11:41:17.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.615+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/202443547 wait complete. 2024-03-29T11:41:17.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.615+0000 7f6ad7d0e640 10 monclient: build_initial_monmap 2024-03-29T11:41:17.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.615+0000 7f6ad7d0e640 10 monclient: monmap: 2024-03-29T11:41:17.620 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:17.620 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:17.620 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:41:17.620 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:41:17.620 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:17.620 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:17.620 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:17.620 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:41:17.620 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.616+0000 7f6ad7d0e640 1 librados: starting msgr at 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.616+0000 7f6ad7d0e640 1 librados: starting objecter 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.616+0000 7f6ad7d0e640 1 Processor -- start 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.616+0000 7f6ad7d0e640 1 -- start start 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.616+0000 7f6ad7d0e640 1 librados: setting wanted keys 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.616+0000 7f6ad7d0e640 1 librados: calling monclient init 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.616+0000 7f6ad7d0e640 10 monclient: init 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.617+0000 7f6ad7d0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.617+0000 7f6ad7d0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.617+0000 7f6ad7d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f6ad00af900 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.617+0000 7f6ad7d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f6ad00b4100 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.617+0000 7f6ad7d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f6ad013ee90 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.617+0000 7f6ad7d0e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad01dfc20 con 0x7f6ad00af900 2024-03-29T11:41:17.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.617+0000 7f6ad7d0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad0138e40 con 0x7f6ad013ee90 2024-03-29T11:41:17.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.617+0000 7f6ad7d0e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad0139120 con 0x7f6ad00b4100 2024-03-29T11:41:17.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.617+0000 7f6ad7d0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:17.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.617+0000 7f6ad7d0e640 10 monclient(hunting): authenticate will time out at 1257.066380s 2024-03-29T11:41:17.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.618+0000 7f6ad62ab640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6ad013ee90 0x7f6ad01df510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:48378/0 (socket says 172.21.15.37:48378) 2024-03-29T11:41:17.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.618+0000 7f6ad62ab640 1 -- 172.21.15.37:0/2771165278 learned_addr learned my addr 172.21.15.37:0/2771165278 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:17.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.619+0000 7f6ac67fc640 10 client.?.objecter ms_handle_connect 0x7f6ad013ee90 2024-03-29T11:41:17.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.619+0000 7f6ac67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:17.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.621+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749676477 0 0) 0x7f6ad0138e40 con 0x7f6ad013ee90 2024-03-29T11:41:17.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.621+0000 7f6ac67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:17.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.621+0000 7f6ac67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:17.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.621+0000 7f6ac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:17.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.621+0000 7f6ac67fc640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:41:17.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.621+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ab4004eb0 con 0x7f6ad013ee90 2024-03-29T11:41:17.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.623+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2549468010 0 0) 0x7f6ab4004eb0 con 0x7f6ad013ee90 2024-03-29T11:41:17.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.623+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ad0138e40 con 0x7f6ad013ee90 2024-03-29T11:41:17.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.623+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6ac80019b0 con 0x7f6ad013ee90 2024-03-29T11:41:17.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.623+0000 7f6ac67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.623+0000 7f6ac67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.623+0000 7f6ac67fc640 10 monclient(hunting): dump: 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.623+0000 7f6ac67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:17.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 64746289 0 0) 0x7f6ad0138e40 con 0x7f6ad013ee90 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 >> v1:172.21.15.37:6790/0 conn(0x7f6ad00b4100 legacy=0x7f6ad01dee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 >> v1:172.21.15.37:6789/0 conn(0x7f6ad00af900 legacy=0x7f6ad0137d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ac67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ac67fc640 1 monclient: found mon.b 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ac67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ac67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad0139400 con 0x7f6ad013ee90 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ac67fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ac67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ac67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ad7d0e640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:41:17.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ad7d0e640 10 monclient: _renew_subs 2024-03-29T11:41:17.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ad7d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:41:17.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/2771165278 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad0138440 con 0x7f6ad013ee90 2024-03-29T11:41:17.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ad7d0e640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:17.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ad7d0e640 10 monclient: _renew_subs 2024-03-29T11:41:17.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ad7d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:41:17.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/2771165278 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad0135fd0 con 0x7f6ad013ee90 2024-03-29T11:41:17.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.624+0000 7f6ad7d0e640 1 librados: init done 2024-03-29T11:41:17.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.625+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ac8004ff0 con 0x7f6ad013ee90 2024-03-29T11:41:17.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.625+0000 7f6ac67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:17.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.625+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6ac8005300 con 0x7f6ad013ee90 2024-03-29T11:41:17.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.625+0000 7f6ac67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:17.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.625+0000 7f6ac67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:17.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.625+0000 7f6ac67fc640 10 monclient: dump: 2024-03-29T11:41:17.631 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:17.631 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:17.631 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:17.631 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:17.631 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:17.631 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:17.631 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:17.631 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:17.631 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:17.631 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:17.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.625+0000 7f6ad7d0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:17.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.625+0000 7f6ad7d0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:17.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.625+0000 7f6ad7d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:41:17.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.626+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/2771165278 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad0075f10 con 0x7f6ad013ee90 2024-03-29T11:41:17.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.626+0000 7f6ac7fff640 4 set_mon_vals no callback set 2024-03-29T11:41:17.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.627+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f6ac801b2d0 con 0x7f6ad013ee90 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.627+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f6ac80850c0 con 0x7f6ad013ee90 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.627+0000 7f6ac67fc640 10 client.4405.objecter ms_dispatch 0x7f6ad0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.628+0000 7f6ac67fc640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.628+0000 7f6ac67fc640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.628+0000 7f6ad5aaa640 10 monclient: get_auth_request con 0x7f6ab406dd70 auth_method 0 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.628+0000 7f6ac67fc640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.628+0000 7f6ac67fc640 10 client.4405.objecter ms_handle_connect 0x7f6ab406dd70 2024-03-29T11:41:17.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.632+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ac80509a0 con 0x7f6ad013ee90 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.632+0000 7f6ac67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.632+0000 7f6ac67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:17.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.976+0000 7f6ad7d0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:17.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.976+0000 7f6ad7d0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:17.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.976+0000 7f6ad7d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:41:17.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.976+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/2771165278 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ad01385d0 con 0x7f6ad013ee90 2024-03-29T11:41:17.979 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:41:17.979 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:17.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.977+0000 7f6ac67fc640 1 -- 172.21.15.37:0/2771165278 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f6ac80528d0 con 0x7f6ad013ee90 2024-03-29T11:41:17.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.977+0000 7f6ac67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:17.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.977+0000 7f6ac67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:17.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.980+0000 7f6ad7d0e640 10 librados: watch_flush enter 2024-03-29T11:41:17.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.981+0000 7f6ad7d0e640 10 librados: watch_flush exit 2024-03-29T11:41:17.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.981+0000 7f6ad7d0e640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-29T11:41:17.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.981+0000 7f6ad7d0e640 10 client.4405.objecter successfully canceled tick 2024-03-29T11:41:17.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.981+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/2771165278 >> v1:172.21.15.146:6816/2899149139 conn(0x7f6ab406dd70 legacy=0x7f6ab4070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.981+0000 7f6ad7d0e640 10 monclient: shutdown 2024-03-29T11:41:17.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.981+0000 7f6ad7d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:17.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.981+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/2771165278 >> v1:172.21.15.146:6789/0 conn(0x7f6ad013ee90 legacy=0x7f6ad01df510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.982+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/2771165278 shutdown_connections 2024-03-29T11:41:17.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.982+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/2771165278 >> 172.21.15.37:0/2771165278 conn(0x7f6ad00b35e0 msgr2=0x7f6ad013a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:17.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.985+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/2771165278 shutdown_connections 2024-03-29T11:41:17.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.985+0000 7f6ad7d0e640 1 -- 172.21.15.37:0/2771165278 wait complete. 2024-03-29T11:41:17.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:17.985+0000 7f6ad7d0e640 1 librados: shutdown 2024-03-29T11:41:18.006 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:24.813 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:24.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.961+0000 7fcb65f32640 10 monclient: get_monmap_and_config 2024-03-29T11:41:24.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.961+0000 7fcb65f32640 10 monclient: build_initial_monmap 2024-03-29T11:41:24.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.961+0000 7fcb65f32640 10 monclient: monmap: 2024-03-29T11:41:24.964 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:24.964 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:24.964 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:41:24.962770+0000 2024-03-29T11:41:24.965 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:41:24.962770+0000 2024-03-29T11:41:24.965 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:24.965 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:24.965 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:24.965 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:41:24.965 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:41:24.965 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:24.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.963+0000 7fcb65f32640 1 Processor -- start 2024-03-29T11:41:24.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.963+0000 7fcb65f32640 1 -- start start 2024-03-29T11:41:24.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.963+0000 7fcb65f32640 10 monclient: init 2024-03-29T11:41:24.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.964+0000 7fcb65f32640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.964+0000 7fcb65f32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.964+0000 7fcb65f32640 10 monclient(hunting): picked mon.noname-a con 0x7fcb601457f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.964+0000 7fcb65f32640 10 monclient(hunting): picked mon.noname-b con 0x7fcb6013f650 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.964+0000 7fcb65f32640 10 monclient(hunting): picked mon.noname-c con 0x7fcb600a9d70 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.964+0000 7fcb65f32640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb600aaef0 con 0x7fcb601457f0 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.964+0000 7fcb65f32640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb60145bd0 con 0x7fcb600a9d70 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.964+0000 7fcb65f32640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb60145eb0 con 0x7fcb6013f650 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.964+0000 7fcb65f32640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.964+0000 7fcb65f32640 10 monclient(hunting): authenticate will time out at 1264.413275s 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.964+0000 7fcb5effd640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fcb6013f650 0x7fcb600b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:44700/0 (socket says 172.21.15.37:44700) 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.964+0000 7fcb5effd640 1 -- 172.21.15.37:0/4033055955 learned_addr learned my addr 172.21.15.37:0/4033055955 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:24.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 1 -- 172.21.15.37:0/4033055955 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475267002 0 0) 0x7fcb600aaef0 con 0x7fcb601457f0 2024-03-29T11:41:24.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 1 -- 172.21.15.37:0/4033055955 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb4c003520 con 0x7fcb601457f0 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 1 -- 172.21.15.37:0/4033055955 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 196726016 0 0) 0x7fcb60145eb0 con 0x7fcb6013f650 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 1 -- 172.21.15.37:0/4033055955 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb4c003840 con 0x7fcb6013f650 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 1 -- 172.21.15.37:0/4033055955 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393521676 0 0) 0x7fcb60145bd0 con 0x7fcb600a9d70 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 1 -- 172.21.15.37:0/4033055955 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb60145eb0 con 0x7fcb600a9d70 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 1 -- 172.21.15.37:0/4033055955 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 235306772 0 0) 0x7fcb4c003840 con 0x7fcb6013f650 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 1 -- 172.21.15.37:0/4033055955 >> v1:172.21.15.146:6789/0 conn(0x7fcb600a9d70 legacy=0x7fcb6014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 1 -- 172.21.15.37:0/4033055955 >> v1:172.21.15.37:6789/0 conn(0x7fcb601457f0 legacy=0x7fcb6013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 1 -- 172.21.15.37:0/4033055955 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb60146340 con 0x7fcb6013f650 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 1 -- 172.21.15.37:0/4033055955 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fcb50001670 con 0x7fcb6013f650 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.965+0000 7fcb5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.966+0000 7fcb5e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.966+0000 7fcb5e7fc640 10 monclient: dump: 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.966+0000 7fcb65f32640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.966+0000 7fcb65f32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.966+0000 7fcb5e7fc640 1 -- 172.21.15.37:0/4033055955 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb500017d0 con 0x7fcb6013f650 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.966+0000 7fcb5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.966+0000 7fcb5e7fc640 1 -- 172.21.15.37:0/4033055955 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fcb50001be0 con 0x7fcb6013f650 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.966+0000 7fcb5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.966+0000 7fcb5e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.966+0000 7fcb5e7fc640 10 monclient: dump: 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:24.973 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:24.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.966+0000 7fcb65f32640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:24.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.966+0000 7fcb65f32640 4 set_mon_vals no callback set 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.967+0000 7fcb65f32640 10 monclient: shutdown 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.967+0000 7fcb65f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.967+0000 7fcb65f32640 1 -- 172.21.15.37:0/4033055955 >> v1:172.21.15.37:6790/0 conn(0x7fcb6013f650 legacy=0x7fcb600b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.967+0000 7fcb65f32640 1 -- 172.21.15.37:0/4033055955 shutdown_connections 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.967+0000 7fcb65f32640 1 -- 172.21.15.37:0/4033055955 >> 172.21.15.37:0/4033055955 conn(0x7fcb60066a00 msgr2=0x7fcb60066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.967+0000 7fcb65f32640 1 -- 172.21.15.37:0/4033055955 shutdown_connections 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.967+0000 7fcb65f32640 1 -- 172.21.15.37:0/4033055955 wait complete. 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.968+0000 7fcb65f32640 10 monclient: build_initial_monmap 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.968+0000 7fcb65f32640 10 monclient: monmap: 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.968+0000 7fcb65f32640 1 librados: starting msgr at 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.968+0000 7fcb65f32640 1 librados: starting objecter 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.968+0000 7fcb65f32640 1 Processor -- start 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.969+0000 7fcb65f32640 1 -- start start 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.969+0000 7fcb65f32640 1 librados: setting wanted keys 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.969+0000 7fcb65f32640 1 librados: calling monclient init 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.969+0000 7fcb65f32640 10 monclient: init 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.969+0000 7fcb65f32640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.969+0000 7fcb65f32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.969+0000 7fcb65f32640 10 monclient(hunting): picked mon.noname-c con 0x7fcb600a9d70 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.969+0000 7fcb65f32640 10 monclient(hunting): picked mon.noname-a con 0x7fcb6013f650 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.970+0000 7fcb65f32640 10 monclient(hunting): picked mon.noname-b con 0x7fcb601457f0 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.970+0000 7fcb65f32640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb600ab9f0 con 0x7fcb6013f650 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.970+0000 7fcb65f32640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb60071dc0 con 0x7fcb601457f0 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.970+0000 7fcb65f32640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb600b0e20 con 0x7fcb600a9d70 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.970+0000 7fcb65f32640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.970+0000 7fcb65f32640 10 monclient(hunting): authenticate will time out at 1264.418991s 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.970+0000 7fcb5effd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fcb6013f650 0x7fcb60070100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55906/0 (socket says 172.21.15.37:55906) 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.970+0000 7fcb5effd640 1 -- 172.21.15.37:0/2092277419 learned_addr learned my addr 172.21.15.37:0/2092277419 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.970+0000 7fcb57fff640 10 client.?.objecter ms_handle_connect 0x7fcb6013f650 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.970+0000 7fcb57fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.970+0000 7fcb57fff640 10 client.?.objecter ms_handle_connect 0x7fcb600a9d70 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.970+0000 7fcb57fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221368689 0 0) 0x7fcb600ab9f0 con 0x7fcb6013f650 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb40004eb0 con 0x7fcb6013f650 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581156216 0 0) 0x7fcb600b0e20 con 0x7fcb600a9d70 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb600ab9f0 con 0x7fcb600a9d70 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 10 client.?.objecter ms_handle_connect 0x7fcb601457f0 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3045405973 0 0) 0x7fcb600ab9f0 con 0x7fcb600a9d70 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb600b0e20 con 0x7fcb600a9d70 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fcb48001810 con 0x7fcb600a9d70 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 10 monclient(hunting): dump: 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.971+0000 7fcb57fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2793439633 0 0) 0x7fcb40004eb0 con 0x7fcb6013f650 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb600ab9f0 con 0x7fcb6013f650 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fcb50001c20 con 0x7fcb6013f650 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 10 monclient(hunting): dump: 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095495856 0 0) 0x7fcb60071dc0 con 0x7fcb601457f0 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb40004eb0 con 0x7fcb601457f0 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3169449196 0 0) 0x7fcb600b0e20 con 0x7fcb600a9d70 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 >> v1:172.21.15.146:6789/0 conn(0x7fcb601457f0 legacy=0x7fcb60070810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 >> v1:172.21.15.37:6789/0 conn(0x7fcb6013f650 legacy=0x7fcb60070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 1 monclient: found mon.c 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb600b0fb0 con 0x7fcb600a9d70 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 10 monclient: _finish_auth 0 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.972+0000 7fcb57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.973+0000 7fcb65f32640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.973+0000 7fcb65f32640 10 monclient: _renew_subs 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.973+0000 7fcb65f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.973+0000 7fcb65f32640 1 -- 172.21.15.37:0/2092277419 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb601324d0 con 0x7fcb600a9d70 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.973+0000 7fcb65f32640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.973+0000 7fcb65f32640 10 monclient: _renew_subs 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.973+0000 7fcb65f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.973+0000 7fcb65f32640 1 -- 172.21.15.37:0/2092277419 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb601e7760 con 0x7fcb600a9d70 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.973+0000 7fcb65f32640 1 librados: init done 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.973+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb48001d00 con 0x7fcb600a9d70 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.973+0000 7fcb57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.973+0000 7fcb5dffb640 4 set_mon_vals no callback set 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.974+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fcb48003790 con 0x7fcb600a9d70 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.974+0000 7fcb57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.974+0000 7fcb57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.974+0000 7fcb57fff640 10 monclient: dump: 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:24.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.974+0000 7fcb65f32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:24.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.975+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fcb48019760 con 0x7fcb600a9d70 2024-03-29T11:41:24.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.976+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fcb48083700 con 0x7fcb600a9d70 2024-03-29T11:41:24.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.976+0000 7fcb57fff640 10 client.4496.objecter ms_dispatch 0x7fcb60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:24.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.976+0000 7fcb57fff640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:24.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.976+0000 7fcb57fff640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:24.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.976+0000 7fcb57fff640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-29T11:41:24.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.976+0000 7fcb65f32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:24.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.976+0000 7fcb65f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:24.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.976+0000 7fcb65f32640 1 -- 172.21.15.37:0/2092277419 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb44005080 con 0x7fcb600a9d70 2024-03-29T11:41:24.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.976+0000 7fcb5effd640 10 monclient: get_auth_request con 0x7fcb4006df00 auth_method 0 2024-03-29T11:41:24.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.977+0000 7fcb57fff640 10 client.4496.objecter ms_handle_connect 0x7fcb4006df00 2024-03-29T11:41:24.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.986+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb4804ee10 con 0x7fcb600a9d70 2024-03-29T11:41:24.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.986+0000 7fcb57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:24.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:24.986+0000 7fcb57fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:25.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.372+0000 7fcb65f32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:25.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.372+0000 7fcb65f32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:25.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.372+0000 7fcb65f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:25.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.372+0000 7fcb65f32640 1 -- 172.21.15.37:0/2092277419 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb44005330 con 0x7fcb600a9d70 2024-03-29T11:41:25.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.378+0000 7fcb57fff640 1 -- 172.21.15.37:0/2092277419 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fcb48050f10 con 0x7fcb600a9d70 2024-03-29T11:41:25.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.378+0000 7fcb57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:25.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.378+0000 7fcb57fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:25.381 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:41:25.381 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:25.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.382+0000 7fcb65f32640 10 librados: watch_flush enter 2024-03-29T11:41:25.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.382+0000 7fcb65f32640 10 librados: watch_flush exit 2024-03-29T11:41:25.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.382+0000 7fcb65f32640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-29T11:41:25.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.382+0000 7fcb65f32640 10 client.4496.objecter successfully canceled tick 2024-03-29T11:41:25.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.382+0000 7fcb65f32640 1 -- 172.21.15.37:0/2092277419 >> v1:172.21.15.146:6816/2899149139 conn(0x7fcb4006df00 legacy=0x7fcb400703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:25.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.382+0000 7fcb65f32640 10 monclient: shutdown 2024-03-29T11:41:25.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.382+0000 7fcb65f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:25.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.382+0000 7fcb65f32640 1 -- 172.21.15.37:0/2092277419 >> v1:172.21.15.37:6790/0 conn(0x7fcb600a9d70 legacy=0x7fcb60142580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:25.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.382+0000 7fcb65f32640 1 -- 172.21.15.37:0/2092277419 shutdown_connections 2024-03-29T11:41:25.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.383+0000 7fcb65f32640 1 -- 172.21.15.37:0/2092277419 >> 172.21.15.37:0/2092277419 conn(0x7fcb60066a00 msgr2=0x7fcb6013bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:25.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.383+0000 7fcb65f32640 1 -- 172.21.15.37:0/2092277419 shutdown_connections 2024-03-29T11:41:25.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.383+0000 7fcb65f32640 1 -- 172.21.15.37:0/2092277419 wait complete. 2024-03-29T11:41:25.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:25.383+0000 7fcb65f32640 1 librados: shutdown 2024-03-29T11:41:25.400 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:32.208 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:32.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.354+0000 7fd8a3fff640 10 monclient: get_monmap_and_config 2024-03-29T11:41:32.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.354+0000 7fd8a3fff640 10 monclient: build_initial_monmap 2024-03-29T11:41:32.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.354+0000 7fd8a3fff640 10 monclient: monmap: 2024-03-29T11:41:32.358 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:32.358 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:32.358 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:41:32.356085+0000 2024-03-29T11:41:32.358 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:41:32.356085+0000 2024-03-29T11:41:32.358 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:32.358 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:32.358 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:32.358 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:41:32.358 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:41:32.358 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:32.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.357+0000 7fd8a3fff640 1 Processor -- start 2024-03-29T11:41:32.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.358+0000 7fd8a3fff640 1 -- start start 2024-03-29T11:41:32.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.358+0000 7fd8a3fff640 10 monclient: init 2024-03-29T11:41:32.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.359+0000 7fd8a3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:32.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.359+0000 7fd8a3fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:32.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.359+0000 7fd8a3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd89c0389e0 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:32.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.359+0000 7fd8a3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd89c038dc0 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:32.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.359+0000 7fd8a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fd89c030660 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:32.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.359+0000 7fd8a3fff640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89c0c54c0 con 0x7fd89c030660 2024-03-29T11:41:32.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.359+0000 7fd8a3fff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89c036430 con 0x7fd89c038dc0 2024-03-29T11:41:32.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.359+0000 7fd8a3fff640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89c039840 con 0x7fd89c0389e0 2024-03-29T11:41:32.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.359+0000 7fd8a3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:32.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.359+0000 7fd8a3fff640 10 monclient(hunting): authenticate will time out at 1271.809418s 2024-03-29T11:41:32.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.362+0000 7fd8a2ffd640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fd89c0389e0 0x7fd89c033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:60219/0 (socket says 172.21.15.37:60219) 2024-03-29T11:41:32.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.362+0000 7fd8a2ffd640 1 -- 172.21.15.37:0/3794611504 learned_addr learned my addr 172.21.15.37:0/3794611504 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:32.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.362+0000 7fd8a1ffb640 1 -- 172.21.15.37:0/3794611504 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842086460 0 0) 0x7fd89c039840 con 0x7fd89c0389e0 2024-03-29T11:41:32.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.362+0000 7fd8a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.362+0000 7fd8a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.362+0000 7fd8a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.362+0000 7fd8a1ffb640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:41:32.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.362+0000 7fd8a1ffb640 1 -- 172.21.15.37:0/3794611504 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd88c003520 con 0x7fd89c0389e0 2024-03-29T11:41:32.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.363+0000 7fd8a1ffb640 1 -- 172.21.15.37:0/3794611504 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1253550397 0 0) 0x7fd88c003520 con 0x7fd89c0389e0 2024-03-29T11:41:32.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.363+0000 7fd8a1ffb640 1 -- 172.21.15.37:0/3794611504 >> v1:172.21.15.146:6789/0 conn(0x7fd89c038dc0 legacy=0x7fd89c0cadd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:41:32.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.363+0000 7fd8a1ffb640 1 -- 172.21.15.37:0/3794611504 >> v1:172.21.15.37:6789/0 conn(0x7fd89c030660 legacy=0x7fd89c0d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:32.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:32.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:41:32.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:32.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:41:32.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a1ffb640 1 -- 172.21.15.37:0/3794611504 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89c035690 con 0x7fd89c0389e0 2024-03-29T11:41:32.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:32.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:32.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:32.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a3fff640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a1ffb640 1 -- 172.21.15.37:0/3794611504 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fd898001690 con 0x7fd89c0389e0 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a1ffb640 10 monclient: dump: 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:32.369 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:32.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:32.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a1ffb640 1 -- 172.21.15.37:0/3794611504 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd898001860 con 0x7fd89c0389e0 2024-03-29T11:41:32.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:32.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a3fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:32.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.364+0000 7fd8a3fff640 4 set_mon_vals no callback set 2024-03-29T11:41:32.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.365+0000 7fd8a1ffb640 1 -- 172.21.15.37:0/3794611504 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fd898001cb0 con 0x7fd89c0389e0 2024-03-29T11:41:32.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.365+0000 7fd8a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.365+0000 7fd8a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:32.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.365+0000 7fd8a1ffb640 10 monclient: dump: 2024-03-29T11:41:32.371 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:32.371 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:32.371 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:32.371 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:32.371 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.372 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:32.372 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:32.372 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:32.372 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:32.372 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:32.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.366+0000 7fd8a3fff640 10 monclient: shutdown 2024-03-29T11:41:32.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.366+0000 7fd8a3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:32.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.366+0000 7fd8a3fff640 1 -- 172.21.15.37:0/3794611504 >> v1:172.21.15.37:6790/0 conn(0x7fd89c0389e0 legacy=0x7fd89c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.366+0000 7fd8a3fff640 1 -- 172.21.15.37:0/3794611504 shutdown_connections 2024-03-29T11:41:32.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.366+0000 7fd8a3fff640 1 -- 172.21.15.37:0/3794611504 >> 172.21.15.37:0/3794611504 conn(0x7fd89c005c00 msgr2=0x7fd89c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:32.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.366+0000 7fd8a3fff640 1 -- 172.21.15.37:0/3794611504 shutdown_connections 2024-03-29T11:41:32.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.366+0000 7fd8a3fff640 1 -- 172.21.15.37:0/3794611504 wait complete. 2024-03-29T11:41:32.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.367+0000 7fd8a3fff640 10 monclient: build_initial_monmap 2024-03-29T11:41:32.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.367+0000 7fd8a3fff640 10 monclient: monmap: 2024-03-29T11:41:32.378 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:32.378 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:32.378 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:41:32.378 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:41:32.378 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:32.378 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:32.378 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:32.378 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:41:32.378 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:41:32.378 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:32.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.367+0000 7fd8a3fff640 1 librados: starting msgr at 2024-03-29T11:41:32.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.367+0000 7fd8a3fff640 1 librados: starting objecter 2024-03-29T11:41:32.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.368+0000 7fd8a3fff640 1 Processor -- start 2024-03-29T11:41:32.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.368+0000 7fd8a3fff640 1 -- start start 2024-03-29T11:41:32.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.368+0000 7fd8a3fff640 1 librados: setting wanted keys 2024-03-29T11:41:32.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.368+0000 7fd8a3fff640 1 librados: calling monclient init 2024-03-29T11:41:32.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.368+0000 7fd8a3fff640 10 monclient: init 2024-03-29T11:41:32.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd8a3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:32.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd8a3fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:32.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd8a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fd89c030660 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:32.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd8a3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd89c0389e0 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:32.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd8a3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd89c00c040 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:32.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd8a3fff640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89c0c0160 con 0x7fd89c030660 2024-03-29T11:41:32.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd8a3fff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89c00d970 con 0x7fd89c0389e0 2024-03-29T11:41:32.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd8a3fff640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89c0c03d0 con 0x7fd89c00c040 2024-03-29T11:41:32.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd8a3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:32.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd8a3fff640 10 monclient(hunting): authenticate will time out at 1271.819138s 2024-03-29T11:41:32.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd8a37fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fd89c00c040 0x7fd89c174040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:52768/0 (socket says 172.21.15.37:52768) 2024-03-29T11:41:32.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd8a37fe640 1 -- 172.21.15.37:0/86021845 learned_addr learned my addr 172.21.15.37:0/86021845 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:32.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 10 client.?.objecter ms_handle_connect 0x7fd89c00c040 2024-03-29T11:41:32.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:32.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 10 client.?.objecter ms_handle_connect 0x7fd89c030660 2024-03-29T11:41:32.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:32.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401045805 0 0) 0x7fd89c0c03d0 con 0x7fd89c00c040 2024-03-29T11:41:32.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:41:32.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd890004eb0 con 0x7fd89c00c040 2024-03-29T11:41:32.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397758871 0 0) 0x7fd89c0c0160 con 0x7fd89c030660 2024-03-29T11:41:32.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:41:32.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd89c0c03d0 con 0x7fd89c030660 2024-03-29T11:41:32.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 10 client.?.objecter ms_handle_connect 0x7fd89c0389e0 2024-03-29T11:41:32.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.369+0000 7fd88b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:32.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 212922254 0 0) 0x7fd890004eb0 con 0x7fd89c00c040 2024-03-29T11:41:32.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd89c0c0160 con 0x7fd89c00c040 2024-03-29T11:41:32.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230582512 0 0) 0x7fd89c00d970 con 0x7fd89c0389e0 2024-03-29T11:41:32.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:41:32.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd890004eb0 con 0x7fd89c0389e0 2024-03-29T11:41:32.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fd894001940 con 0x7fd89c00c040 2024-03-29T11:41:32.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:32.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 10 monclient(hunting): dump: 2024-03-29T11:41:32.382 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:32.382 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:32.382 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:32.382 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:32.382 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3014730737 0 0) 0x7fd89c0c03d0 con 0x7fd89c030660 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd89c00d970 con 0x7fd89c030660 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fd898001b70 con 0x7fd89c030660 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 10 monclient(hunting): dump: 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:32.383 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:32.384 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:32.384 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.384 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:32.384 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:32.384 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:32.384 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:32.384 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:32.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3168659454 0 0) 0x7fd89c0c0160 con 0x7fd89c00c040 2024-03-29T11:41:32.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 >> v1:172.21.15.146:6789/0 conn(0x7fd89c0389e0 legacy=0x7fd89c0c0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 >> v1:172.21.15.37:6789/0 conn(0x7fd89c030660 legacy=0x7fd89c0be720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:32.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 1 monclient: found mon.c 2024-03-29T11:41:32.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:32.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:32.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.370+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89c0c06b0 con 0x7fd89c00c040 2024-03-29T11:41:32.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd88b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:32.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd88b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:32.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd88b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:32.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd8a3fff640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:41:32.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd8a3fff640 10 monclient: _renew_subs 2024-03-29T11:41:32.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd8a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:32.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd8a3fff640 1 -- 172.21.15.37:0/86021845 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd89c0bee30 con 0x7fd89c00c040 2024-03-29T11:41:32.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd8a3fff640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:32.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd8a3fff640 10 monclient: _renew_subs 2024-03-29T11:41:32.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd8a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:32.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd8a3fff640 1 -- 172.21.15.37:0/86021845 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd89c175760 con 0x7fd89c00c040 2024-03-29T11:41:32.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd8a3fff640 1 librados: init done 2024-03-29T11:41:32.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8940035f0 con 0x7fd89c00c040 2024-03-29T11:41:32.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd88b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:32.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fd8940039c0 con 0x7fd89c00c040 2024-03-29T11:41:32.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd88b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd88b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:32.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd88b7fe640 10 monclient: dump: 2024-03-29T11:41:32.386 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:32.386 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:32.386 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:32.386 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:32.386 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.386 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:32.387 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:32.387 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:32.387 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:32.387 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:32.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.371+0000 7fd8a0ff9640 4 set_mon_vals no callback set 2024-03-29T11:41:32.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.373+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fd894003ba0 con 0x7fd89c00c040 2024-03-29T11:41:32.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.373+0000 7fd8a3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:32.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.373+0000 7fd8a3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:32.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.373+0000 7fd8a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:32.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.373+0000 7fd8a3fff640 1 -- 172.21.15.37:0/86021845 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd884005080 con 0x7fd89c00c040 2024-03-29T11:41:32.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.374+0000 7fd8a2ffd640 10 monclient: get_auth_request con 0x7fd89006de60 auth_method 0 2024-03-29T11:41:32.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.374+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fd894084a10 con 0x7fd89c00c040 2024-03-29T11:41:32.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.374+0000 7fd88b7fe640 10 client.4502.objecter ms_dispatch 0x7fd8a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:32.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.374+0000 7fd88b7fe640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:32.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.374+0000 7fd88b7fe640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:32.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.374+0000 7fd88b7fe640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-29T11:41:32.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.374+0000 7fd88b7fe640 10 client.4502.objecter ms_handle_connect 0x7fd89006de60 2024-03-29T11:41:32.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.383+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd894050350 con 0x7fd89c00c040 2024-03-29T11:41:32.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.383+0000 7fd88b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:32.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.383+0000 7fd88b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:32.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.780+0000 7fd8a3fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:32.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.780+0000 7fd8a3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:32.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.780+0000 7fd8a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:41:32.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.780+0000 7fd8a3fff640 1 -- 172.21.15.37:0/86021845 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd884005330 con 0x7fd89c00c040 2024-03-29T11:41:32.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.781+0000 7fd88b7fe640 1 -- 172.21.15.37:0/86021845 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fd8940840c0 con 0x7fd89c00c040 2024-03-29T11:41:32.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.781+0000 7fd88b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:32.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.781+0000 7fd88b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:32.785 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:41:32.785 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:32.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.785+0000 7fd8a3fff640 10 librados: watch_flush enter 2024-03-29T11:41:32.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.785+0000 7fd8a3fff640 10 librados: watch_flush exit 2024-03-29T11:41:32.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.785+0000 7fd8a3fff640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-29T11:41:32.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.785+0000 7fd8a3fff640 10 client.4502.objecter successfully canceled tick 2024-03-29T11:41:32.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.785+0000 7fd8a3fff640 1 -- 172.21.15.37:0/86021845 >> v1:172.21.15.146:6816/2899149139 conn(0x7fd89006de60 legacy=0x7fd890070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.785+0000 7fd8a3fff640 10 monclient: shutdown 2024-03-29T11:41:32.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.785+0000 7fd8a3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:32.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.785+0000 7fd8a3fff640 1 -- 172.21.15.37:0/86021845 >> v1:172.21.15.37:6790/0 conn(0x7fd89c00c040 legacy=0x7fd89c174040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.785+0000 7fd8a3fff640 1 -- 172.21.15.37:0/86021845 shutdown_connections 2024-03-29T11:41:32.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.785+0000 7fd8a3fff640 1 -- 172.21.15.37:0/86021845 >> 172.21.15.37:0/86021845 conn(0x7fd89c005c00 msgr2=0x7fd89c034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:32.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.786+0000 7fd8a3fff640 1 -- 172.21.15.37:0/86021845 shutdown_connections 2024-03-29T11:41:32.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.786+0000 7fd8a3fff640 1 -- 172.21.15.37:0/86021845 wait complete. 2024-03-29T11:41:32.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:32.786+0000 7fd8a3fff640 1 librados: shutdown 2024-03-29T11:41:32.804 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:39.611 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:39.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.756+0000 7f87633f4640 10 monclient: get_monmap_and_config 2024-03-29T11:41:39.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.756+0000 7f87633f4640 10 monclient: build_initial_monmap 2024-03-29T11:41:39.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.757+0000 7f87633f4640 10 monclient: monmap: 2024-03-29T11:41:39.761 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:39.761 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:39.761 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:41:39.758881+0000 2024-03-29T11:41:39.761 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:41:39.758881+0000 2024-03-29T11:41:39.761 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:39.761 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:39.761 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:39.761 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:41:39.761 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:41:39.761 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:39.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.757+0000 7f87633f4640 1 Processor -- start 2024-03-29T11:41:39.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.758+0000 7f87633f4640 1 -- start start 2024-03-29T11:41:39.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.758+0000 7f87633f4640 10 monclient: init 2024-03-29T11:41:39.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.758+0000 7f87633f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:39.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.758+0000 7f87633f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:39.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.758+0000 7f87633f4640 10 monclient(hunting): picked mon.noname-a con 0x7f875c0b0770 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:39.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.759+0000 7f87633f4640 10 monclient(hunting): picked mon.noname-c con 0x7f875c14a700 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:39.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.759+0000 7f87633f4640 10 monclient(hunting): picked mon.noname-b con 0x7f875c0ac5a0 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:39.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.759+0000 7f87633f4640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875c0aba70 con 0x7f875c0b0770 2024-03-29T11:41:39.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.759+0000 7f87633f4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875c0aecb0 con 0x7f875c14a700 2024-03-29T11:41:39.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.759+0000 7f87633f4640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875c0af150 con 0x7f875c0ac5a0 2024-03-29T11:41:39.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.759+0000 7f87633f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:39.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.759+0000 7f87633f4640 10 monclient(hunting): authenticate will time out at 1279.208982s 2024-03-29T11:41:39.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.760+0000 7f8761190640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f875c0b0770 0x7f875c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47794/0 (socket says 172.21.15.37:47794) 2024-03-29T11:41:39.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.760+0000 7f8761190640 1 -- 172.21.15.37:0/2353646324 learned_addr learned my addr 172.21.15.37:0/2353646324 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:39.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.761+0000 7f8753fff640 1 -- 172.21.15.37:0/2353646324 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131153419 0 0) 0x7f875c0aba70 con 0x7f875c0b0770 2024-03-29T11:41:39.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.761+0000 7f8753fff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:39.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.761+0000 7f8753fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:39.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.762+0000 7f8753fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:39.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.762+0000 7f8753fff640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:41:39.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.762+0000 7f8753fff640 1 -- 172.21.15.37:0/2353646324 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8748003520 con 0x7f875c0b0770 2024-03-29T11:41:39.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.762+0000 7f8753fff640 1 -- 172.21.15.37:0/2353646324 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321906617 0 0) 0x7f875c0af150 con 0x7f875c0ac5a0 2024-03-29T11:41:39.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.762+0000 7f8753fff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:39.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.762+0000 7f8753fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:39.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.762+0000 7f8753fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:39.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.762+0000 7f8753fff640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:41:39.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.762+0000 7f8753fff640 1 -- 172.21.15.37:0/2353646324 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f875c0aba70 con 0x7f875c0ac5a0 2024-03-29T11:41:39.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.762+0000 7f8753fff640 1 -- 172.21.15.37:0/2353646324 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 588410635 0 0) 0x7f8748003520 con 0x7f875c0b0770 2024-03-29T11:41:39.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.762+0000 7f8753fff640 1 -- 172.21.15.37:0/2353646324 >> v1:172.21.15.37:6790/0 conn(0x7f875c0ac5a0 legacy=0x7f875c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:39.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.762+0000 7f8753fff640 1 -- 172.21.15.37:0/2353646324 >> v1:172.21.15.146:6789/0 conn(0x7f875c14a700 legacy=0x7f875c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.763+0000 7f8753fff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.763+0000 7f8753fff640 1 monclient: found mon.noname-a 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.763+0000 7f8753fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.763+0000 7f8753fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.763+0000 7f8753fff640 1 -- 172.21.15.37:0/2353646324 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c0ac9d0 con 0x7f875c0b0770 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.763+0000 7f8753fff640 10 monclient: _finish_auth 0 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.763+0000 7f8753fff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.763+0000 7f8753fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.763+0000 7f8753fff640 1 -- 172.21.15.37:0/2353646324 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8754001a10 con 0x7f875c0b0770 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.763+0000 7f8753fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.763+0000 7f8753fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.763+0000 7f8753fff640 10 monclient: dump: 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:39.769 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:39.770 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:39.770 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:39.770 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:39.770 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:39.770 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:39.770 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:39.770 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:39.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.764+0000 7f87633f4640 5 monclient: authenticate success, global_id 4452 2024-03-29T11:41:39.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.765+0000 7f87633f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:39.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.766+0000 7f8753fff640 1 -- 172.21.15.37:0/2353646324 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8754001b70 con 0x7f875c0b0770 2024-03-29T11:41:39.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.766+0000 7f8753fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:39.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.766+0000 7f87633f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:39.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.766+0000 7f87633f4640 4 set_mon_vals no callback set 2024-03-29T11:41:39.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.766+0000 7f8753fff640 1 -- 172.21.15.37:0/2353646324 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8754001f40 con 0x7f875c0b0770 2024-03-29T11:41:39.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.766+0000 7f8753fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:39.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.766+0000 7f8753fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:39.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.766+0000 7f8753fff640 10 monclient: dump: 2024-03-29T11:41:39.773 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:39.773 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:39.773 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:39.773 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:39.773 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:39.774 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:39.774 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:39.774 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:39.774 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:39.774 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:39.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.767+0000 7f87633f4640 10 monclient: shutdown 2024-03-29T11:41:39.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.767+0000 7f87633f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:39.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.767+0000 7f87633f4640 1 -- 172.21.15.37:0/2353646324 >> v1:172.21.15.37:6789/0 conn(0x7f875c0b0770 legacy=0x7f875c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:39.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.767+0000 7f87633f4640 1 -- 172.21.15.37:0/2353646324 shutdown_connections 2024-03-29T11:41:39.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.767+0000 7f87633f4640 1 -- 172.21.15.37:0/2353646324 >> 172.21.15.37:0/2353646324 conn(0x7f875c06b660 msgr2=0x7f875c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:39.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.767+0000 7f87633f4640 1 -- 172.21.15.37:0/2353646324 shutdown_connections 2024-03-29T11:41:39.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.767+0000 7f87633f4640 1 -- 172.21.15.37:0/2353646324 wait complete. 2024-03-29T11:41:39.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.768+0000 7f87633f4640 10 monclient: build_initial_monmap 2024-03-29T11:41:39.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.768+0000 7f87633f4640 10 monclient: monmap: 2024-03-29T11:41:39.775 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:39.775 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:39.775 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:41:39.775 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:41:39.775 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:39.775 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:39.775 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:39.775 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:41:39.775 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:41:39.775 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:39.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.769+0000 7f87633f4640 1 librados: starting msgr at 2024-03-29T11:41:39.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.769+0000 7f87633f4640 1 librados: starting objecter 2024-03-29T11:41:39.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.769+0000 7f87633f4640 1 Processor -- start 2024-03-29T11:41:39.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.769+0000 7f87633f4640 1 -- start start 2024-03-29T11:41:39.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.769+0000 7f87633f4640 1 librados: setting wanted keys 2024-03-29T11:41:39.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.769+0000 7f87633f4640 1 librados: calling monclient init 2024-03-29T11:41:39.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.769+0000 7f87633f4640 10 monclient: init 2024-03-29T11:41:39.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.770+0000 7f87633f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:39.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.770+0000 7f87633f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:39.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.770+0000 7f87633f4640 10 monclient(hunting): picked mon.noname-b con 0x7f875c0ac5a0 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:39.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.770+0000 7f87633f4640 10 monclient(hunting): picked mon.noname-c con 0x7f875c0b0770 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:39.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.770+0000 7f87633f4640 10 monclient(hunting): picked mon.noname-a con 0x7f875c14a700 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:39.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.770+0000 7f87633f4640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875c12a2f0 con 0x7f875c14a700 2024-03-29T11:41:39.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.770+0000 7f87633f4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875c1db790 con 0x7f875c0ac5a0 2024-03-29T11:41:39.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.770+0000 7f87633f4640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875c125ba0 con 0x7f875c0b0770 2024-03-29T11:41:39.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.770+0000 7f87633f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:39.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.770+0000 7f87633f4640 10 monclient(hunting): authenticate will time out at 1279.220148s 2024-03-29T11:41:39.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.773+0000 7f8761190640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f875c0ac5a0 0x7f875c1dbc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:53121/0 (socket says 172.21.15.37:53121) 2024-03-29T11:41:39.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.773+0000 7f8761190640 1 -- 172.21.15.37:0/956797668 learned_addr learned my addr 172.21.15.37:0/956797668 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:39.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.773+0000 7f8751ffb640 10 client.?.objecter ms_handle_connect 0x7f875c0ac5a0 2024-03-29T11:41:39.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.773+0000 7f8751ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:39.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.774+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801844311 0 0) 0x7f875c1db790 con 0x7f875c0ac5a0 2024-03-29T11:41:39.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.774+0000 7f8751ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:39.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.774+0000 7f8751ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:39.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.774+0000 7f8751ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:39.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.774+0000 7f8751ffb640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:41:39.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.774+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8740004eb0 con 0x7f875c0ac5a0 2024-03-29T11:41:39.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.774+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2056800790 0 0) 0x7f8740004eb0 con 0x7f875c0ac5a0 2024-03-29T11:41:39.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.774+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f875c1db790 con 0x7f875c0ac5a0 2024-03-29T11:41:39.781 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.774+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8754001f40 con 0x7f875c0ac5a0 2024-03-29T11:41:39.781 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.774+0000 7f8751ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:39.781 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.774+0000 7f8751ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:39.781 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.774+0000 7f8751ffb640 10 monclient(hunting): dump: 2024-03-29T11:41:39.781 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:39.781 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:39.781 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:39.781 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:39.781 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:39.781 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:39.781 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:39.781 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:39.781 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:39.782 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:39.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.774+0000 7f8751ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:39.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 10 client.?.objecter ms_handle_connect 0x7f875c14a700 2024-03-29T11:41:39.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:39.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1168231233 0 0) 0x7f875c1db790 con 0x7f875c0ac5a0 2024-03-29T11:41:39.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 >> v1:172.21.15.37:6790/0 conn(0x7f875c0b0770 legacy=0x7f875c1471c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:39.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 >> v1:172.21.15.37:6789/0 conn(0x7f875c14a700 legacy=0x7f875c1e7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:39.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:39.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 1 monclient: found mon.b 2024-03-29T11:41:39.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:39.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:41:39.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c12a540 con 0x7f875c0ac5a0 2024-03-29T11:41:39.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 10 client.?.objecter ms_handle_connect 0x7f875c0b0770 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.775+0000 7f8751ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.776+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8754004e10 con 0x7f875c0ac5a0 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.776+0000 7f8751ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.776+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f87540051e0 con 0x7f875c0ac5a0 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.776+0000 7f8751ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.776+0000 7f8751ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.776+0000 7f8751ffb640 10 monclient: dump: 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:39.783 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:39.784 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:39.784 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:39.784 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:39.784 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:39.784 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:39.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.777+0000 7f8752ffd640 4 set_mon_vals no callback set 2024-03-29T11:41:39.786 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.777+0000 7f87633f4640 5 monclient: authenticate success, global_id 4420 2024-03-29T11:41:39.786 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.777+0000 7f87633f4640 10 monclient: _renew_subs 2024-03-29T11:41:39.786 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.777+0000 7f87633f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:41:39.786 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.777+0000 7f87633f4640 1 -- 172.21.15.37:0/956797668 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f875c128150 con 0x7f875c0ac5a0 2024-03-29T11:41:39.786 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.778+0000 7f87633f4640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:39.786 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.778+0000 7f87633f4640 10 monclient: _renew_subs 2024-03-29T11:41:39.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.778+0000 7f87633f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:41:39.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.778+0000 7f87633f4640 1 -- 172.21.15.37:0/956797668 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f875c128950 con 0x7f875c0ac5a0 2024-03-29T11:41:39.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.778+0000 7f87633f4640 1 librados: init done 2024-03-29T11:41:39.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.780+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f8754005340 con 0x7f875c0ac5a0 2024-03-29T11:41:39.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.780+0000 7f87633f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:39.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.780+0000 7f87633f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:39.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.780+0000 7f87633f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:41:39.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.780+0000 7f87633f4640 1 -- 172.21.15.37:0/956797668 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f875c0ab560 con 0x7f875c0ac5a0 2024-03-29T11:41:39.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.780+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f8754084fa0 con 0x7f875c0ac5a0 2024-03-29T11:41:39.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.780+0000 7f8751ffb640 10 client.4420.objecter ms_dispatch 0x7f875c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:39.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.780+0000 7f8751ffb640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:39.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.780+0000 7f8751ffb640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:39.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.780+0000 7f8751ffb640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-29T11:41:39.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.781+0000 7f876098f640 10 monclient: get_auth_request con 0x7f874006dcf0 auth_method 0 2024-03-29T11:41:39.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.783+0000 7f8751ffb640 10 client.4420.objecter ms_handle_connect 0x7f874006dcf0 2024-03-29T11:41:39.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.792+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8754050880 con 0x7f875c0ac5a0 2024-03-29T11:41:39.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.792+0000 7f8751ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:39.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:39.792+0000 7f8751ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:40.133 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.130+0000 7f87633f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:40.133 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.130+0000 7f87633f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:40.134 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.130+0000 7f87633f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:41:40.134 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.130+0000 7f87633f4640 1 -- 172.21.15.37:0/956797668 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f875c075f10 con 0x7f875c0ac5a0 2024-03-29T11:41:40.134 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.131+0000 7f8751ffb640 1 -- 172.21.15.37:0/956797668 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f87540527b0 con 0x7f875c0ac5a0 2024-03-29T11:41:40.137 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:41:40.137 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:40.137 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.131+0000 7f8751ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:40.137 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.131+0000 7f8751ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:40.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.136+0000 7f87633f4640 10 librados: watch_flush enter 2024-03-29T11:41:40.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.137+0000 7f87633f4640 10 librados: watch_flush exit 2024-03-29T11:41:40.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.137+0000 7f87633f4640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-29T11:41:40.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.137+0000 7f87633f4640 10 client.4420.objecter successfully canceled tick 2024-03-29T11:41:40.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.137+0000 7f87633f4640 1 -- 172.21.15.37:0/956797668 >> v1:172.21.15.146:6816/2899149139 conn(0x7f874006dcf0 legacy=0x7f87400701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:40.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.137+0000 7f87633f4640 10 monclient: shutdown 2024-03-29T11:41:40.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.137+0000 7f87633f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:40.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.137+0000 7f87633f4640 1 -- 172.21.15.37:0/956797668 >> v1:172.21.15.146:6789/0 conn(0x7f875c0ac5a0 legacy=0x7f875c1dbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:40.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.138+0000 7f87633f4640 1 -- 172.21.15.37:0/956797668 shutdown_connections 2024-03-29T11:41:40.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.139+0000 7f87633f4640 1 -- 172.21.15.37:0/956797668 >> 172.21.15.37:0/956797668 conn(0x7f875c06b660 msgr2=0x7f875c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:40.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.139+0000 7f87633f4640 1 -- 172.21.15.37:0/956797668 shutdown_connections 2024-03-29T11:41:40.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.139+0000 7f87633f4640 1 -- 172.21.15.37:0/956797668 wait complete. 2024-03-29T11:41:40.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:40.139+0000 7f87633f4640 1 librados: shutdown 2024-03-29T11:41:40.159 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:46.966 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:47.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.112+0000 7fdc73b3e640 10 monclient: get_monmap_and_config 2024-03-29T11:41:47.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.112+0000 7fdc73b3e640 10 monclient: build_initial_monmap 2024-03-29T11:41:47.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.112+0000 7fdc73b3e640 10 monclient: monmap: 2024-03-29T11:41:47.115 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:47.115 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:47.115 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:41:47.113467+0000 2024-03-29T11:41:47.115 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:41:47.113467+0000 2024-03-29T11:41:47.116 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:47.116 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:47.116 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:47.116 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:41:47.116 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:41:47.116 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:47.116 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.113+0000 7fdc73b3e640 1 Processor -- start 2024-03-29T11:41:47.116 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.113+0000 7fdc73b3e640 1 -- start start 2024-03-29T11:41:47.116 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.113+0000 7fdc73b3e640 10 monclient: init 2024-03-29T11:41:47.119 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.114+0000 7fdc73b3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:47.119 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.114+0000 7fdc73b3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:47.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.114+0000 7fdc73b3e640 10 monclient(hunting): picked mon.noname-a con 0x7fdc6c0b1f60 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:47.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.114+0000 7fdc73b3e640 10 monclient(hunting): picked mon.noname-b con 0x7fdc6c071eb0 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:47.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.114+0000 7fdc73b3e640 10 monclient(hunting): picked mon.noname-c con 0x7fdc6c143d10 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:47.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.114+0000 7fdc73b3e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c06ef80 con 0x7fdc6c0b1f60 2024-03-29T11:41:47.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.114+0000 7fdc73b3e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c06f260 con 0x7fdc6c143d10 2024-03-29T11:41:47.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.114+0000 7fdc73b3e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c0b40c0 con 0x7fdc6c071eb0 2024-03-29T11:41:47.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.114+0000 7fdc73b3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:47.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.114+0000 7fdc73b3e640 10 monclient(hunting): authenticate will time out at 1286.563686s 2024-03-29T11:41:47.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.117+0000 7fdc710d9640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fdc6c071eb0 0x7fdc6c146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:60525/0 (socket says 172.21.15.37:60525) 2024-03-29T11:41:47.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.117+0000 7fdc710d9640 1 -- 172.21.15.37:0/2263137796 learned_addr learned my addr 172.21.15.37:0/2263137796 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:47.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.118+0000 7fdc708d8640 1 -- 172.21.15.37:0/2263137796 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009312868 0 0) 0x7fdc6c0b40c0 con 0x7fdc6c071eb0 2024-03-29T11:41:47.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.118+0000 7fdc708d8640 10 monclient(hunting): _negotiate 2024-03-29T11:41:47.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.118+0000 7fdc708d8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:47.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.118+0000 7fdc708d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:47.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.118+0000 7fdc708d8640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:41:47.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.119+0000 7fdc708d8640 1 -- 172.21.15.37:0/2263137796 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc58003520 con 0x7fdc6c071eb0 2024-03-29T11:41:47.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.119+0000 7fdc708d8640 1 -- 172.21.15.37:0/2263137796 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086104178 0 0) 0x7fdc6c06ef80 con 0x7fdc6c0b1f60 2024-03-29T11:41:47.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.119+0000 7fdc708d8640 10 monclient(hunting): _negotiate 2024-03-29T11:41:47.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.119+0000 7fdc708d8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:47.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.119+0000 7fdc708d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:47.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.119+0000 7fdc708d8640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:41:47.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.119+0000 7fdc708d8640 1 -- 172.21.15.37:0/2263137796 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc6c0b40c0 con 0x7fdc6c0b1f60 2024-03-29T11:41:47.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.119+0000 7fdc708d8640 1 -- 172.21.15.37:0/2263137796 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 676592758 0 0) 0x7fdc58003520 con 0x7fdc6c071eb0 2024-03-29T11:41:47.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.119+0000 7fdc708d8640 1 -- 172.21.15.37:0/2263137796 >> v1:172.21.15.146:6789/0 conn(0x7fdc6c143d10 legacy=0x7fdc6c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.119+0000 7fdc708d8640 1 -- 172.21.15.37:0/2263137796 >> v1:172.21.15.37:6789/0 conn(0x7fdc6c0b1f60 legacy=0x7fdc6c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.119+0000 7fdc708d8640 10 monclient: _finish_hunting 0 2024-03-29T11:41:47.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc708d8640 1 monclient: found mon.noname-b 2024-03-29T11:41:47.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc708d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:47.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc708d8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:41:47.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc708d8640 1 -- 172.21.15.37:0/2263137796 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c155d70 con 0x7fdc6c071eb0 2024-03-29T11:41:47.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc708d8640 10 monclient: _finish_auth 0 2024-03-29T11:41:47.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc708d8640 10 monclient: _check_auth_tickets 2024-03-29T11:41:47.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc708d8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc73b3e640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc73b3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc708d8640 1 -- 172.21.15.37:0/2263137796 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fdc68001610 con 0x7fdc6c071eb0 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc708d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc708d8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc708d8640 10 monclient: dump: 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:47.127 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc73b3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc708d8640 1 -- 172.21.15.37:0/2263137796 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc68001830 con 0x7fdc6c071eb0 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc708d8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc73b3e640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.120+0000 7fdc73b3e640 4 set_mon_vals no callback set 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.121+0000 7fdc708d8640 1 -- 172.21.15.37:0/2263137796 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fdc68001c00 con 0x7fdc6c071eb0 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.121+0000 7fdc708d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.121+0000 7fdc708d8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.121+0000 7fdc708d8640 10 monclient: dump: 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.128 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:47.129 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:47.129 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:47.129 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:47.129 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:47.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.121+0000 7fdc73b3e640 10 monclient: shutdown 2024-03-29T11:41:47.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.121+0000 7fdc73b3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:47.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.121+0000 7fdc73b3e640 1 -- 172.21.15.37:0/2263137796 >> v1:172.21.15.37:6790/0 conn(0x7fdc6c071eb0 legacy=0x7fdc6c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.121+0000 7fdc73b3e640 1 -- 172.21.15.37:0/2263137796 shutdown_connections 2024-03-29T11:41:47.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.121+0000 7fdc73b3e640 1 -- 172.21.15.37:0/2263137796 >> 172.21.15.37:0/2263137796 conn(0x7fdc6c066a00 msgr2=0x7fdc6c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:47.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.122+0000 7fdc73b3e640 1 -- 172.21.15.37:0/2263137796 shutdown_connections 2024-03-29T11:41:47.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.122+0000 7fdc73b3e640 1 -- 172.21.15.37:0/2263137796 wait complete. 2024-03-29T11:41:47.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.125+0000 7fdc73b3e640 10 monclient: build_initial_monmap 2024-03-29T11:41:47.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.125+0000 7fdc73b3e640 10 monclient: monmap: 2024-03-29T11:41:47.131 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:47.131 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:47.131 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:41:47.131 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:41:47.132 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:47.132 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:47.132 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:47.132 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:41:47.132 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:41:47.132 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:47.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.125+0000 7fdc73b3e640 1 librados: starting msgr at 2024-03-29T11:41:47.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.125+0000 7fdc73b3e640 1 librados: starting objecter 2024-03-29T11:41:47.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.126+0000 7fdc73b3e640 1 Processor -- start 2024-03-29T11:41:47.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.126+0000 7fdc73b3e640 1 -- start start 2024-03-29T11:41:47.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.126+0000 7fdc73b3e640 1 librados: setting wanted keys 2024-03-29T11:41:47.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.126+0000 7fdc73b3e640 1 librados: calling monclient init 2024-03-29T11:41:47.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.126+0000 7fdc73b3e640 10 monclient: init 2024-03-29T11:41:47.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.127+0000 7fdc73b3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:47.133 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.127+0000 7fdc73b3e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:47.133 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.127+0000 7fdc73b3e640 10 monclient(hunting): picked mon.noname-b con 0x7fdc6c0b1f60 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:47.133 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.127+0000 7fdc73b3e640 10 monclient(hunting): picked mon.noname-a con 0x7fdc6c143d10 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:47.133 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.127+0000 7fdc73b3e640 10 monclient(hunting): picked mon.noname-c con 0x7fdc6c136e70 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:47.133 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.128+0000 7fdc73b3e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c06f6d0 con 0x7fdc6c143d10 2024-03-29T11:41:47.133 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.130+0000 7fdc73b3e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c070e80 con 0x7fdc6c0b1f60 2024-03-29T11:41:47.133 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.130+0000 7fdc73b3e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c071160 con 0x7fdc6c136e70 2024-03-29T11:41:47.134 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.130+0000 7fdc73b3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:47.134 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.130+0000 7fdc73b3e640 10 monclient(hunting): authenticate will time out at 1286.579645s 2024-03-29T11:41:47.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.131+0000 7fdc710d9640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fdc6c143d10 0x7fdc6c153770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47066/0 (socket says 172.21.15.37:47066) 2024-03-29T11:41:47.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.131+0000 7fdc710d9640 1 -- 172.21.15.37:0/4213973151 learned_addr learned my addr 172.21.15.37:0/4213973151 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:47.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 10 client.?.objecter ms_handle_connect 0x7fdc6c143d10 2024-03-29T11:41:47.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:47.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498671752 0 0) 0x7fdc6c06f6d0 con 0x7fdc6c143d10 2024-03-29T11:41:47.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:47.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:47.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:47.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:41:47.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc50004eb0 con 0x7fdc6c143d10 2024-03-29T11:41:47.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 244663461 0 0) 0x7fdc50004eb0 con 0x7fdc6c143d10 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc6c06f6d0 con 0x7fdc6c143d10 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fdc68001b80 con 0x7fdc6c143d10 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 10 monclient(hunting): dump: 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:47.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.132+0000 7fdc627fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:47.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 10 client.?.objecter ms_handle_connect 0x7fdc6c136e70 2024-03-29T11:41:47.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:47.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 10 client.?.objecter ms_handle_connect 0x7fdc6c0b1f60 2024-03-29T11:41:47.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:47.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1111795160 0 0) 0x7fdc6c06f6d0 con 0x7fdc6c143d10 2024-03-29T11:41:47.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 >> v1:172.21.15.37:6790/0 conn(0x7fdc6c136e70 legacy=0x7fdc6c153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 >> v1:172.21.15.146:6789/0 conn(0x7fdc6c0b1f60 legacy=0x7fdc6c131c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:47.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 1 monclient: found mon.a 2024-03-29T11:41:47.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:47.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c142740 con 0x7fdc6c143d10 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc73b3e640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc73b3e640 10 monclient: _renew_subs 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc73b3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc73b3e640 1 -- 172.21.15.37:0/4213973151 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6c06f920 con 0x7fdc6c143d10 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc73b3e640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc73b3e640 10 monclient: _renew_subs 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc73b3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc73b3e640 1 -- 172.21.15.37:0/4213973151 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6c132380 con 0x7fdc6c143d10 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc73b3e640 1 librados: init done 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc68004b00 con 0x7fdc6c143d10 2024-03-29T11:41:47.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fdc68004ed0 con 0x7fdc6c143d10 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.135+0000 7fdc627fc640 10 monclient: dump: 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.136+0000 7fdc63fff640 4 set_mon_vals no callback set 2024-03-29T11:41:47.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.138+0000 7fdc73b3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:47.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.138+0000 7fdc73b3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:47.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.138+0000 7fdc73b3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:41:47.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.138+0000 7fdc73b3e640 1 -- 172.21.15.37:0/4213973151 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc54005080 con 0x7fdc6c143d10 2024-03-29T11:41:47.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.138+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fdc68005030 con 0x7fdc6c143d10 2024-03-29T11:41:47.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.139+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fdc68084c00 con 0x7fdc6c143d10 2024-03-29T11:41:47.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.139+0000 7fdc627fc640 10 client.4461.objecter ms_dispatch 0x7fdc6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:47.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.139+0000 7fdc627fc640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:47.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.139+0000 7fdc627fc640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:47.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.139+0000 7fdc627fc640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-29T11:41:47.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.139+0000 7fdc718da640 10 monclient: get_auth_request con 0x7fdc5006dc50 auth_method 0 2024-03-29T11:41:47.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.140+0000 7fdc627fc640 10 client.4461.objecter ms_handle_connect 0x7fdc5006dc50 2024-03-29T11:41:47.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.146+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc680504e0 con 0x7fdc6c143d10 2024-03-29T11:41:47.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.146+0000 7fdc627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:47.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.146+0000 7fdc627fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:47.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.473+0000 7fdc73b3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:47.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.473+0000 7fdc73b3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:47.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.473+0000 7fdc73b3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:41:47.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.473+0000 7fdc73b3e640 1 -- 172.21.15.37:0/4213973151 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc54005330 con 0x7fdc6c143d10 2024-03-29T11:41:47.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.474+0000 7fdc627fc640 1 -- 172.21.15.37:0/4213973151 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fdc68052410 con 0x7fdc6c143d10 2024-03-29T11:41:47.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.475+0000 7fdc627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:47.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.475+0000 7fdc627fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:47.478 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:41:47.478 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:47.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.478+0000 7fdc73b3e640 10 librados: watch_flush enter 2024-03-29T11:41:47.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.480+0000 7fdc73b3e640 10 librados: watch_flush exit 2024-03-29T11:41:47.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.480+0000 7fdc73b3e640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-29T11:41:47.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.480+0000 7fdc73b3e640 10 client.4461.objecter successfully canceled tick 2024-03-29T11:41:47.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.480+0000 7fdc73b3e640 1 -- 172.21.15.37:0/4213973151 >> v1:172.21.15.146:6816/2899149139 conn(0x7fdc5006dc50 legacy=0x7fdc50070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.480+0000 7fdc73b3e640 10 monclient: shutdown 2024-03-29T11:41:47.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.480+0000 7fdc73b3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:47.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.480+0000 7fdc73b3e640 1 -- 172.21.15.37:0/4213973151 >> v1:172.21.15.37:6789/0 conn(0x7fdc6c143d10 legacy=0x7fdc6c153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.481+0000 7fdc73b3e640 1 -- 172.21.15.37:0/4213973151 shutdown_connections 2024-03-29T11:41:47.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.481+0000 7fdc73b3e640 1 -- 172.21.15.37:0/4213973151 >> 172.21.15.37:0/4213973151 conn(0x7fdc6c066a00 msgr2=0x7fdc6c146960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:47.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.481+0000 7fdc73b3e640 1 -- 172.21.15.37:0/4213973151 shutdown_connections 2024-03-29T11:41:47.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.481+0000 7fdc73b3e640 1 -- 172.21.15.37:0/4213973151 wait complete. 2024-03-29T11:41:47.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:47.482+0000 7fdc73b3e640 1 librados: shutdown 2024-03-29T11:41:47.501 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:54.307 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:54.457 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.455+0000 7fbdf47c6640 10 monclient: get_monmap_and_config 2024-03-29T11:41:54.458 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.455+0000 7fbdf47c6640 10 monclient: build_initial_monmap 2024-03-29T11:41:54.458 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.455+0000 7fbdf47c6640 10 monclient: monmap: 2024-03-29T11:41:54.458 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:54.458 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:54.458 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:41:54.456081+0000 2024-03-29T11:41:54.458 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:41:54.456081+0000 2024-03-29T11:41:54.458 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:54.458 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:54.458 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:54.458 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:41:54.459 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:41:54.459 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:54.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.458+0000 7fbdf47c6640 1 Processor -- start 2024-03-29T11:41:54.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.458+0000 7fbdf47c6640 1 -- start start 2024-03-29T11:41:54.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.458+0000 7fbdf47c6640 10 monclient: init 2024-03-29T11:41:54.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.459+0000 7fbdf47c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:54.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.459+0000 7fbdf47c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:54.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.459+0000 7fbdf47c6640 10 monclient(hunting): picked mon.noname-b con 0x7fbdec1457f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:54.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.459+0000 7fbdf47c6640 10 monclient(hunting): picked mon.noname-a con 0x7fbdec13f650 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:54.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.459+0000 7fbdf47c6640 10 monclient(hunting): picked mon.noname-c con 0x7fbdec0a9d70 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:54.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.460+0000 7fbdf47c6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec0aaef0 con 0x7fbdec13f650 2024-03-29T11:41:54.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.460+0000 7fbdf47c6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec145bd0 con 0x7fbdec0a9d70 2024-03-29T11:41:54.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.460+0000 7fbdf47c6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec145eb0 con 0x7fbdec1457f0 2024-03-29T11:41:54.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.460+0000 7fbdf47c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:54.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.460+0000 7fbdf47c6640 10 monclient(hunting): authenticate will time out at 1293.908977s 2024-03-29T11:41:54.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.460+0000 7fbdf2562640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fbdec1457f0 0x7fbdec13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:41974/0 (socket says 172.21.15.37:41974) 2024-03-29T11:41:54.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.460+0000 7fbdf2562640 1 -- 172.21.15.37:0/2474387830 learned_addr learned my addr 172.21.15.37:0/2474387830 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:54.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.461+0000 7fbdf1560640 1 -- 172.21.15.37:0/2474387830 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930508733 0 0) 0x7fbdec145eb0 con 0x7fbdec1457f0 2024-03-29T11:41:54.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.461+0000 7fbdf1560640 10 monclient(hunting): _negotiate 2024-03-29T11:41:54.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.461+0000 7fbdf1560640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:54.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.461+0000 7fbdf1560640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:54.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.461+0000 7fbdf1560640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:41:54.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.461+0000 7fbdf1560640 1 -- 172.21.15.37:0/2474387830 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd8003520 con 0x7fbdec1457f0 2024-03-29T11:41:54.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.461+0000 7fbdf1560640 1 -- 172.21.15.37:0/2474387830 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2763493694 0 0) 0x7fbdd8003520 con 0x7fbdec1457f0 2024-03-29T11:41:54.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.462+0000 7fbdf1560640 1 -- 172.21.15.37:0/2474387830 >> v1:172.21.15.146:6789/0 conn(0x7fbdec0a9d70 legacy=0x7fbdec14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:54.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.465+0000 7fbdf1560640 1 -- 172.21.15.37:0/2474387830 >> v1:172.21.15.37:6789/0 conn(0x7fbdec13f650 legacy=0x7fbdec0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.465+0000 7fbdf1560640 10 monclient: _finish_hunting 0 2024-03-29T11:41:54.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.465+0000 7fbdf1560640 1 monclient: found mon.noname-b 2024-03-29T11:41:54.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.465+0000 7fbdf1560640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:54.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.465+0000 7fbdf1560640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:41:54.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.465+0000 7fbdf1560640 1 -- 172.21.15.37:0/2474387830 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec146340 con 0x7fbdec1457f0 2024-03-29T11:41:54.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.465+0000 7fbdf1560640 10 monclient: _finish_auth 0 2024-03-29T11:41:54.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.465+0000 7fbdf1560640 10 monclient: _check_auth_tickets 2024-03-29T11:41:54.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.465+0000 7fbdf1560640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:54.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.465+0000 7fbdf1560640 1 -- 172.21.15.37:0/2474387830 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbde8001690 con 0x7fbdec1457f0 2024-03-29T11:41:54.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.465+0000 7fbdf1560640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.465+0000 7fbdf1560640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:54.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.465+0000 7fbdf1560640 10 monclient: dump: 2024-03-29T11:41:54.469 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:54.469 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:54.469 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:54.470 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:54.470 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.470 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:54.470 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:54.470 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:54.470 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:54.470 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:54.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.466+0000 7fbdf1560640 1 -- 172.21.15.37:0/2474387830 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbde8001860 con 0x7fbdec1457f0 2024-03-29T11:41:54.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.466+0000 7fbdf1560640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:54.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.466+0000 7fbdf1560640 1 -- 172.21.15.37:0/2474387830 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbde8001c70 con 0x7fbdec1457f0 2024-03-29T11:41:54.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.466+0000 7fbdf1560640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.466+0000 7fbdf1560640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:54.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.466+0000 7fbdf1560640 10 monclient: dump: 2024-03-29T11:41:54.473 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:54.473 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:54.473 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:54.473 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:54.473 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.473 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:54.473 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:54.473 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:54.473 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:54.473 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:54.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.467+0000 7fbdf47c6640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:41:54.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.467+0000 7fbdf47c6640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:54.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.467+0000 7fbdf47c6640 4 set_mon_vals no callback set 2024-03-29T11:41:54.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.468+0000 7fbdf47c6640 10 monclient: shutdown 2024-03-29T11:41:54.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.468+0000 7fbdf47c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:54.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.468+0000 7fbdf47c6640 1 -- 172.21.15.37:0/2474387830 >> v1:172.21.15.37:6790/0 conn(0x7fbdec1457f0 legacy=0x7fbdec13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.468+0000 7fbdf47c6640 1 -- 172.21.15.37:0/2474387830 shutdown_connections 2024-03-29T11:41:54.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.468+0000 7fbdf47c6640 1 -- 172.21.15.37:0/2474387830 >> 172.21.15.37:0/2474387830 conn(0x7fbdec066a00 msgr2=0x7fbdec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:54.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.468+0000 7fbdf47c6640 1 -- 172.21.15.37:0/2474387830 shutdown_connections 2024-03-29T11:41:54.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.468+0000 7fbdf47c6640 1 -- 172.21.15.37:0/2474387830 wait complete. 2024-03-29T11:41:54.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.469+0000 7fbdf47c6640 10 monclient: build_initial_monmap 2024-03-29T11:41:54.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.469+0000 7fbdf47c6640 10 monclient: monmap: 2024-03-29T11:41:54.474 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:41:54.474 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:54.474 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:41:54.475 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:41:54.475 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:54.475 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:54.475 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:41:54.475 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:41:54.475 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:41:54.475 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:54.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.469+0000 7fbdf47c6640 1 librados: starting msgr at 2024-03-29T11:41:54.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.469+0000 7fbdf47c6640 1 librados: starting objecter 2024-03-29T11:41:54.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.470+0000 7fbdf47c6640 1 Processor -- start 2024-03-29T11:41:54.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.470+0000 7fbdf47c6640 1 -- start start 2024-03-29T11:41:54.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.470+0000 7fbdf47c6640 1 librados: setting wanted keys 2024-03-29T11:41:54.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.470+0000 7fbdf47c6640 1 librados: calling monclient init 2024-03-29T11:41:54.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.470+0000 7fbdf47c6640 10 monclient: init 2024-03-29T11:41:54.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.470+0000 7fbdf47c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.470+0000 7fbdf47c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.470+0000 7fbdf47c6640 10 monclient(hunting): picked mon.noname-b con 0x7fbdec0a9d70 addr v1:172.21.15.146:6789/0 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.470+0000 7fbdf47c6640 10 monclient(hunting): picked mon.noname-c con 0x7fbdec13f650 addr v1:172.21.15.37:6790/0 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.472+0000 7fbdf47c6640 10 monclient(hunting): picked mon.noname-a con 0x7fbdec1457f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.472+0000 7fbdf47c6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec137a20 con 0x7fbdec1457f0 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.472+0000 7fbdf47c6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec0ac110 con 0x7fbdec0a9d70 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.472+0000 7fbdf47c6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec0ac3f0 con 0x7fbdec13f650 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.472+0000 7fbdf47c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.472+0000 7fbdf47c6640 10 monclient(hunting): authenticate will time out at 1293.921112s 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.474+0000 7fbdf1d61640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fbdec13f650 0x7fbdec136c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:49344/0 (socket says 172.21.15.37:49344) 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.474+0000 7fbdf1d61640 1 -- 172.21.15.37:0/4223168079 learned_addr learned my addr 172.21.15.37:0/4223168079 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 10 client.?.objecter ms_handle_connect 0x7fbdec13f650 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 10 client.?.objecter ms_handle_connect 0x7fbdec1457f0 2024-03-29T11:41:54.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:54.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815043793 0 0) 0x7fbdec137a20 con 0x7fbdec1457f0 2024-03-29T11:41:54.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:54.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:54.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:54.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:41:54.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd0004eb0 con 0x7fbdec1457f0 2024-03-29T11:41:54.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543823765 0 0) 0x7fbdec0ac3f0 con 0x7fbdec13f650 2024-03-29T11:41:54.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:54.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:54.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:54.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:41:54.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.475+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdec137a20 con 0x7fbdec13f650 2024-03-29T11:41:54.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.476+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3174197960 0 0) 0x7fbdd0004eb0 con 0x7fbdec1457f0 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.476+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdec0ac3f0 con 0x7fbdec1457f0 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.476+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbddc0017f0 con 0x7fbdec1457f0 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.476+0000 7fbde2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.476+0000 7fbde2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.476+0000 7fbde2ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:54.483 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:54.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.476+0000 7fbde2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:54.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.476+0000 7fbde2ffd640 10 client.?.objecter ms_handle_connect 0x7fbdec0a9d70 2024-03-29T11:41:54.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.476+0000 7fbde2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:54.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 522938495 0 0) 0x7fbdec0ac3f0 con 0x7fbdec1457f0 2024-03-29T11:41:54.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 >> v1:172.21.15.37:6790/0 conn(0x7fbdec13f650 legacy=0x7fbdec136c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 >> v1:172.21.15.146:6789/0 conn(0x7fbdec0a9d70 legacy=0x7fbdec142580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbde2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:54.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbde2ffd640 1 monclient: found mon.a 2024-03-29T11:41:54.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbde2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:54.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbde2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:41:54.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec0ac6d0 con 0x7fbdec1457f0 2024-03-29T11:41:54.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbde2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:54.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbde2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:54.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbde2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:54.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbdf47c6640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:41:54.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbdf47c6640 10 monclient: _renew_subs 2024-03-29T11:41:54.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbdf47c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:41:54.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbdf47c6640 1 -- 172.21.15.37:0/4223168079 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdec144da0 con 0x7fbdec1457f0 2024-03-29T11:41:54.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbdf47c6640 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbdf47c6640 10 monclient: _renew_subs 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbdf47c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbdf47c6640 1 -- 172.21.15.37:0/4223168079 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdec13fda0 con 0x7fbdec1457f0 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.477+0000 7fbdf47c6640 1 librados: init done 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.478+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbddc001d00 con 0x7fbdec1457f0 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.478+0000 7fbde2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.478+0000 7fbdf0d5f640 4 set_mon_vals no callback set 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.479+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbddc003830 con 0x7fbdec1457f0 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.479+0000 7fbde2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.479+0000 7fbde2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.479+0000 7fbde2ffd640 10 monclient: dump: 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:54.488 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:41:54.489 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.489 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:41:54.489 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:41:54.489 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:41:54.489 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:41:54.489 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:41:54.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.480+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fbddc019800 con 0x7fbdec1457f0 2024-03-29T11:41:54.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.480+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fbddc083810 con 0x7fbdec1457f0 2024-03-29T11:41:54.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.480+0000 7fbde2ffd640 10 client.4467.objecter ms_dispatch 0x7fbdec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:54.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.480+0000 7fbde2ffd640 3 client.4467.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:54.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.480+0000 7fbde2ffd640 3 client.4467.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:54.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.481+0000 7fbde2ffd640 20 client.4467.objecter dump_active .. 0 homeless 2024-03-29T11:41:54.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.481+0000 7fbdf47c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:54.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.481+0000 7fbdf2562640 10 monclient: get_auth_request con 0x7fbdd006dda0 auth_method 0 2024-03-29T11:41:54.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.481+0000 7fbdf47c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:54.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.481+0000 7fbdf47c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:41:54.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.481+0000 7fbdf47c6640 1 -- 172.21.15.37:0/4223168079 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd4005080 con 0x7fbdec1457f0 2024-03-29T11:41:54.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.481+0000 7fbde2ffd640 10 client.4467.objecter ms_handle_connect 0x7fbdd006dda0 2024-03-29T11:41:54.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.490+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbddc04ef20 con 0x7fbdec1457f0 2024-03-29T11:41:54.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.490+0000 7fbde2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:54.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.490+0000 7fbde2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:54.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.817+0000 7fbdf47c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:54.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.817+0000 7fbdf47c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:54.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.817+0000 7fbdf47c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:41:54.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.817+0000 7fbdf47c6640 1 -- 172.21.15.37:0/4223168079 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdd4005330 con 0x7fbdec1457f0 2024-03-29T11:41:54.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.819+0000 7fbde2ffd640 1 -- 172.21.15.37:0/4223168079 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fbddc051020 con 0x7fbdec1457f0 2024-03-29T11:41:54.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.819+0000 7fbde2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:54.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.819+0000 7fbde2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:54.823 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:41:54.823 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:54.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.823+0000 7fbdf47c6640 10 librados: watch_flush enter 2024-03-29T11:41:54.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.823+0000 7fbdf47c6640 10 librados: watch_flush exit 2024-03-29T11:41:54.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.823+0000 7fbdf47c6640 20 client.4467.objecter shutdown clearing up homeless session... 2024-03-29T11:41:54.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.823+0000 7fbdf47c6640 10 client.4467.objecter successfully canceled tick 2024-03-29T11:41:54.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.823+0000 7fbdf47c6640 1 -- 172.21.15.37:0/4223168079 >> v1:172.21.15.146:6816/2899149139 conn(0x7fbdd006dda0 legacy=0x7fbdd0070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.823+0000 7fbdf47c6640 10 monclient: shutdown 2024-03-29T11:41:54.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.823+0000 7fbdf47c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:54.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.823+0000 7fbdf47c6640 1 -- 172.21.15.37:0/4223168079 >> v1:172.21.15.37:6789/0 conn(0x7fbdec1457f0 legacy=0x7fbdec137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.823+0000 7fbdf47c6640 1 -- 172.21.15.37:0/4223168079 shutdown_connections 2024-03-29T11:41:54.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.823+0000 7fbdf47c6640 1 -- 172.21.15.37:0/4223168079 >> 172.21.15.37:0/4223168079 conn(0x7fbdec066a00 msgr2=0x7fbdec066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:54.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.824+0000 7fbdf47c6640 1 -- 172.21.15.37:0/4223168079 shutdown_connections 2024-03-29T11:41:54.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.824+0000 7fbdf47c6640 1 -- 172.21.15.37:0/4223168079 wait complete. 2024-03-29T11:41:54.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:41:54.824+0000 7fbdf47c6640 1 librados: shutdown 2024-03-29T11:41:54.844 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:01.651 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:01.798 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.795+0000 7f5dc7857640 10 monclient: get_monmap_and_config 2024-03-29T11:42:01.798 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.795+0000 7f5dc7857640 10 monclient: build_initial_monmap 2024-03-29T11:42:01.798 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.795+0000 7f5dc7857640 10 monclient: monmap: 2024-03-29T11:42:01.798 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:01.799 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:01.799 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:42:01.796382+0000 2024-03-29T11:42:01.799 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:42:01.796382+0000 2024-03-29T11:42:01.799 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:01.799 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:01.799 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:01.799 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:42:01.799 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:42:01.799 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:01.800 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.796+0000 7f5dc7857640 1 Processor -- start 2024-03-29T11:42:01.800 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.796+0000 7f5dc7857640 1 -- start start 2024-03-29T11:42:01.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.796+0000 7f5dc7857640 10 monclient: init 2024-03-29T11:42:01.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.797+0000 7f5dc7857640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:01.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.797+0000 7f5dc7857640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:01.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.797+0000 7f5dc7857640 10 monclient(hunting): picked mon.noname-b con 0x7f5dc00b0860 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:01.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.797+0000 7f5dc7857640 10 monclient(hunting): picked mon.noname-a con 0x7f5dc013ef30 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:01.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.797+0000 7f5dc7857640 10 monclient(hunting): picked mon.noname-c con 0x7f5dc00af8e0 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:01.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.797+0000 7f5dc7857640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc00b0c40 con 0x7f5dc013ef30 2024-03-29T11:42:01.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.797+0000 7f5dc7857640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc00aed10 con 0x7f5dc00af8e0 2024-03-29T11:42:01.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.797+0000 7f5dc7857640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc00afd50 con 0x7f5dc00b0860 2024-03-29T11:42:01.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.797+0000 7f5dc7857640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:01.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.797+0000 7f5dc7857640 10 monclient(hunting): authenticate will time out at 1301.246481s 2024-03-29T11:42:01.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.799+0000 7f5dc55f3640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f5dc00b0860 0x7f5dc013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:41320/0 (socket says 172.21.15.37:41320) 2024-03-29T11:42:01.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.799+0000 7f5dc55f3640 1 -- 172.21.15.37:0/597868685 learned_addr learned my addr 172.21.15.37:0/597868685 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:01.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.802+0000 7f5db7fff640 1 -- 172.21.15.37:0/597868685 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614316783 0 0) 0x7f5dc00afd50 con 0x7f5dc00b0860 2024-03-29T11:42:01.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.802+0000 7f5db7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:01.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.802+0000 7f5db7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:01.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.802+0000 7f5db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:01.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.802+0000 7f5db7fff640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:42:01.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.802+0000 7f5db7fff640 1 -- 172.21.15.37:0/597868685 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dac003520 con 0x7f5dc00b0860 2024-03-29T11:42:01.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.802+0000 7f5db7fff640 1 -- 172.21.15.37:0/597868685 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835334604 0 0) 0x7f5dc00b0c40 con 0x7f5dc013ef30 2024-03-29T11:42:01.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.802+0000 7f5db7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:01.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.802+0000 7f5db7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:01.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.802+0000 7f5db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:01.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.802+0000 7f5db7fff640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:42:01.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 1 -- 172.21.15.37:0/597868685 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc00afd50 con 0x7f5dc013ef30 2024-03-29T11:42:01.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 1 -- 172.21.15.37:0/597868685 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152177359 0 0) 0x7f5dc00aed10 con 0x7f5dc00af8e0 2024-03-29T11:42:01.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:01.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:01.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:01.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:42:01.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 1 -- 172.21.15.37:0/597868685 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc00b0c40 con 0x7f5dc00af8e0 2024-03-29T11:42:01.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 1 -- 172.21.15.37:0/597868685 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4246878436 0 0) 0x7f5dc00afd50 con 0x7f5dc013ef30 2024-03-29T11:42:01.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 1 -- 172.21.15.37:0/597868685 >> v1:172.21.15.37:6790/0 conn(0x7f5dc00b0860 legacy=0x7f5dc013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:01.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 1 -- 172.21.15.37:0/597868685 >> v1:172.21.15.146:6789/0 conn(0x7f5dc00af8e0 legacy=0x7f5dc0149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:01.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:01.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 1 monclient: found mon.noname-a 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 1 -- 172.21.15.37:0/597868685 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc00ab1b0 con 0x7f5dc013ef30 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.803+0000 7f5db7fff640 10 monclient: _finish_auth 0 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.804+0000 7f5db7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.804+0000 7f5db7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.804+0000 7f5db7fff640 1 -- 172.21.15.37:0/597868685 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5db0001520 con 0x7f5dc013ef30 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.804+0000 7f5db7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.804+0000 7f5db7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.804+0000 7f5db7fff640 10 monclient: dump: 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:01.808 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.804+0000 7f5db7fff640 1 -- 172.21.15.37:0/597868685 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5db0001680 con 0x7f5dc013ef30 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.804+0000 7f5db7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.804+0000 7f5db7fff640 1 -- 172.21.15.37:0/597868685 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5db0001a90 con 0x7f5dc013ef30 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.804+0000 7f5db7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.804+0000 7f5db7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.804+0000 7f5db7fff640 10 monclient: dump: 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:01.809 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:01.810 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:01.810 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:01.810 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:01.810 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:01.810 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:01.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.806+0000 7f5dc7857640 5 monclient: authenticate success, global_id 4470 2024-03-29T11:42:01.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.806+0000 7f5dc7857640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:01.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.806+0000 7f5dc7857640 4 set_mon_vals no callback set 2024-03-29T11:42:01.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.807+0000 7f5dc7857640 10 monclient: shutdown 2024-03-29T11:42:01.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.807+0000 7f5dc7857640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:01.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.807+0000 7f5dc7857640 1 -- 172.21.15.37:0/597868685 >> v1:172.21.15.37:6789/0 conn(0x7f5dc013ef30 legacy=0x7f5dc00af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:01.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.807+0000 7f5dc7857640 1 -- 172.21.15.37:0/597868685 shutdown_connections 2024-03-29T11:42:01.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.807+0000 7f5dc7857640 1 -- 172.21.15.37:0/597868685 >> 172.21.15.37:0/597868685 conn(0x7f5dc006b660 msgr2=0x7f5dc006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:01.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.808+0000 7f5dc7857640 1 -- 172.21.15.37:0/597868685 shutdown_connections 2024-03-29T11:42:01.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.808+0000 7f5dc7857640 1 -- 172.21.15.37:0/597868685 wait complete. 2024-03-29T11:42:01.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.809+0000 7f5dc7857640 10 monclient: build_initial_monmap 2024-03-29T11:42:01.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.810+0000 7f5dc7857640 10 monclient: monmap: 2024-03-29T11:42:01.812 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:01.812 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:01.812 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:42:01.812 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:42:01.812 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:01.812 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:01.812 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:01.812 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:42:01.813 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:42:01.813 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:01.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.810+0000 7f5dc7857640 1 librados: starting msgr at 2024-03-29T11:42:01.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.810+0000 7f5dc7857640 1 librados: starting objecter 2024-03-29T11:42:01.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.810+0000 7f5dc7857640 1 Processor -- start 2024-03-29T11:42:01.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.810+0000 7f5dc7857640 1 -- start start 2024-03-29T11:42:01.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.810+0000 7f5dc7857640 1 librados: setting wanted keys 2024-03-29T11:42:01.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.810+0000 7f5dc7857640 1 librados: calling monclient init 2024-03-29T11:42:01.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.810+0000 7f5dc7857640 10 monclient: init 2024-03-29T11:42:01.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.811+0000 7f5dc7857640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:01.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.811+0000 7f5dc7857640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:01.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.812+0000 7f5dc7857640 10 monclient(hunting): picked mon.noname-b con 0x7f5dc00af8e0 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:01.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.812+0000 7f5dc7857640 10 monclient(hunting): picked mon.noname-c con 0x7f5dc00b0860 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:01.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.812+0000 7f5dc7857640 10 monclient(hunting): picked mon.noname-a con 0x7f5dc013afd0 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:01.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.812+0000 7f5dc7857640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc00b38c0 con 0x7f5dc013afd0 2024-03-29T11:42:01.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.812+0000 7f5dc7857640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc00b35c0 con 0x7f5dc00af8e0 2024-03-29T11:42:01.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.812+0000 7f5dc7857640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc0135e20 con 0x7f5dc00b0860 2024-03-29T11:42:01.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.812+0000 7f5dc4df2640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f5dc00b0860 0x7f5dc01d32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:43420/0 (socket says 172.21.15.37:43420) 2024-03-29T11:42:01.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.812+0000 7f5dc4df2640 1 -- 172.21.15.37:0/1725916740 learned_addr learned my addr 172.21.15.37:0/1725916740 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:01.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.812+0000 7f5dc7857640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:01.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.812+0000 7f5dc7857640 10 monclient(hunting): authenticate will time out at 1301.261399s 2024-03-29T11:42:01.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.814+0000 7f5db5ffb640 10 client.?.objecter ms_handle_connect 0x7f5dc00b0860 2024-03-29T11:42:01.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.814+0000 7f5db5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:01.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.814+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761156539 0 0) 0x7f5dc0135e20 con 0x7f5dc00b0860 2024-03-29T11:42:01.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.814+0000 7f5db5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:01.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.814+0000 7f5db5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:01.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.814+0000 7f5db5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:01.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.814+0000 7f5db5ffb640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.814+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da4004eb0 con 0x7f5dc00b0860 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4004256668 0 0) 0x7f5da4004eb0 con 0x7f5dc00b0860 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dc0135e20 con 0x7f5dc00b0860 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5db00019d0 con 0x7f5dc00b0860 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:01.820 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:01.821 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:01.821 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:01.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:01.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3253305629 0 0) 0x7f5dc0135e20 con 0x7f5dc00b0860 2024-03-29T11:42:01.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 >> v1:172.21.15.146:6789/0 conn(0x7f5dc00af8e0 legacy=0x7f5dc00b2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:01.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 >> v1:172.21.15.37:6789/0 conn(0x7f5dc013afd0 legacy=0x7f5dc01ddd00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:01.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:01.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 1 monclient: found mon.c 2024-03-29T11:42:01.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:01.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:01.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc01d39d0 con 0x7f5dc00b0860 2024-03-29T11:42:01.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:01.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:01.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5db5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:01.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5dc7857640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:42:01.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5dc7857640 10 monclient: _renew_subs 2024-03-29T11:42:01.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5dc7857640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:01.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.815+0000 7f5dc7857640 1 -- 172.21.15.37:0/1725916740 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc01d3c20 con 0x7f5dc00b0860 2024-03-29T11:42:01.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.816+0000 7f5dc7857640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:01.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.816+0000 7f5dc7857640 10 monclient: _renew_subs 2024-03-29T11:42:01.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.816+0000 7f5dc7857640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:01.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.816+0000 7f5dc7857640 1 -- 172.21.15.37:0/1725916740 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc01df600 con 0x7f5dc00b0860 2024-03-29T11:42:01.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.816+0000 7f5dc7857640 1 librados: init done 2024-03-29T11:42:01.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.818+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5db0001b30 con 0x7f5dc00b0860 2024-03-29T11:42:01.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.818+0000 7f5db5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:01.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.818+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5db0003b00 con 0x7f5dc00b0860 2024-03-29T11:42:01.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.818+0000 7f5db5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:01.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.818+0000 7f5db5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:01.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.818+0000 7f5db5ffb640 10 monclient: dump: 2024-03-29T11:42:01.825 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:01.825 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:01.825 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:01.825 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:01.825 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:01.825 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:01.825 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:01.825 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:01.825 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:01.825 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:01.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.818+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f5db0003d20 con 0x7f5dc00b0860 2024-03-29T11:42:01.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.818+0000 7f5db77fe640 4 set_mon_vals no callback set 2024-03-29T11:42:01.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.819+0000 7f5dc55f3640 10 monclient: get_auth_request con 0x7f5da406dd20 auth_method 0 2024-03-29T11:42:01.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.819+0000 7f5dc7857640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:01.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.819+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f5db0084bb0 con 0x7f5dc00b0860 2024-03-29T11:42:01.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.819+0000 7f5db5ffb640 10 client.4520.objecter ms_dispatch 0x7f5dc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:01.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.819+0000 7f5dc7857640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:01.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.819+0000 7f5db5ffb640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:01.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.819+0000 7f5db5ffb640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:01.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.819+0000 7f5dc7857640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:01.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.819+0000 7f5dc7857640 1 -- 172.21.15.37:0/1725916740 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da0005080 con 0x7f5dc00b0860 2024-03-29T11:42:01.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.820+0000 7f5db5ffb640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-29T11:42:01.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.820+0000 7f5db5ffb640 10 client.4520.objecter ms_handle_connect 0x7f5da406dd20 2024-03-29T11:42:01.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.828+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5db00505f0 con 0x7f5dc00b0860 2024-03-29T11:42:01.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.828+0000 7f5db5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:01.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:01.828+0000 7f5db5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:02.165 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:42:02.165 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:02.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.161+0000 7f5dc7857640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:02.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.161+0000 7f5dc7857640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:02.166 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.161+0000 7f5dc7857640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:02.166 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.161+0000 7f5dc7857640 1 -- 172.21.15.37:0/1725916740 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5da0005330 con 0x7f5dc00b0860 2024-03-29T11:42:02.166 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.162+0000 7f5db5ffb640 1 -- 172.21.15.37:0/1725916740 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f5db0052520 con 0x7f5dc00b0860 2024-03-29T11:42:02.166 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.162+0000 7f5db5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:02.166 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.162+0000 7f5db5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:02.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.166+0000 7f5dc7857640 10 librados: watch_flush enter 2024-03-29T11:42:02.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.166+0000 7f5dc7857640 10 librados: watch_flush exit 2024-03-29T11:42:02.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.166+0000 7f5dc7857640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-29T11:42:02.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.166+0000 7f5dc7857640 10 client.4520.objecter successfully canceled tick 2024-03-29T11:42:02.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.166+0000 7f5dc7857640 1 -- 172.21.15.37:0/1725916740 >> v1:172.21.15.146:6816/2899149139 conn(0x7f5da406dd20 legacy=0x7f5da40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:02.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.166+0000 7f5dc7857640 10 monclient: shutdown 2024-03-29T11:42:02.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.166+0000 7f5dc7857640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:02.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.166+0000 7f5dc7857640 1 -- 172.21.15.37:0/1725916740 >> v1:172.21.15.37:6790/0 conn(0x7f5dc00b0860 legacy=0x7f5dc01d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:02.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.166+0000 7f5dc7857640 1 -- 172.21.15.37:0/1725916740 shutdown_connections 2024-03-29T11:42:02.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.166+0000 7f5dc7857640 1 -- 172.21.15.37:0/1725916740 >> 172.21.15.37:0/1725916740 conn(0x7f5dc006b660 msgr2=0x7f5dc01c8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:02.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.170+0000 7f5dc7857640 1 -- 172.21.15.37:0/1725916740 shutdown_connections 2024-03-29T11:42:02.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.170+0000 7f5dc7857640 1 -- 172.21.15.37:0/1725916740 wait complete. 2024-03-29T11:42:02.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:02.171+0000 7f5dc7857640 1 librados: shutdown 2024-03-29T11:42:02.191 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:08.998 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:09.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.142+0000 7f016f5c1640 10 monclient: get_monmap_and_config 2024-03-29T11:42:09.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.142+0000 7f016f5c1640 10 monclient: build_initial_monmap 2024-03-29T11:42:09.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.142+0000 7f016f5c1640 10 monclient: monmap: 2024-03-29T11:42:09.145 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:09.145 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:09.145 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:42:09.142911+0000 2024-03-29T11:42:09.145 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:42:09.142911+0000 2024-03-29T11:42:09.145 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:09.145 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:09.145 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:09.145 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:42:09.145 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:42:09.146 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:09.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.143+0000 7f016f5c1640 1 Processor -- start 2024-03-29T11:42:09.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.143+0000 7f016f5c1640 1 -- start start 2024-03-29T11:42:09.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.143+0000 7f016f5c1640 10 monclient: init 2024-03-29T11:42:09.148 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.144+0000 7f016f5c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:09.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.144+0000 7f016f5c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:09.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.144+0000 7f016f5c1640 10 monclient(hunting): picked mon.noname-c con 0x7f01681457f0 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:09.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.144+0000 7f016f5c1640 10 monclient(hunting): picked mon.noname-b con 0x7f016813f650 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:09.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.144+0000 7f016f5c1640 10 monclient(hunting): picked mon.noname-a con 0x7f01680a9d70 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:09.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.144+0000 7f016f5c1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01680aaef0 con 0x7f01680a9d70 2024-03-29T11:42:09.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.144+0000 7f016f5c1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0168145bd0 con 0x7f01681457f0 2024-03-29T11:42:09.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.144+0000 7f016f5c1640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0168145eb0 con 0x7f016813f650 2024-03-29T11:42:09.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.144+0000 7f016f5c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:09.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.144+0000 7f016f5c1640 10 monclient(hunting): authenticate will time out at 1308.593890s 2024-03-29T11:42:09.152 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.147+0000 7f016d35d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f01681457f0 0x7f016813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:49685/0 (socket says 172.21.15.37:49685) 2024-03-29T11:42:09.152 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.147+0000 7f016d35d640 1 -- 172.21.15.37:0/1890733922 learned_addr learned my addr 172.21.15.37:0/1890733922 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:09.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.150+0000 7f015ffff640 1 -- 172.21.15.37:0/1890733922 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48261018 0 0) 0x7f01680aaef0 con 0x7f01680a9d70 2024-03-29T11:42:09.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.150+0000 7f015ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:09.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.150+0000 7f015ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:09.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.150+0000 7f015ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:09.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.150+0000 7f015ffff640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:42:09.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.150+0000 7f015ffff640 1 -- 172.21.15.37:0/1890733922 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0154003520 con 0x7f01680a9d70 2024-03-29T11:42:09.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 1 -- 172.21.15.37:0/1890733922 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277893981 0 0) 0x7f0168145eb0 con 0x7f016813f650 2024-03-29T11:42:09.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:09.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:09.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:09.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:42:09.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 1 -- 172.21.15.37:0/1890733922 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01680aaef0 con 0x7f016813f650 2024-03-29T11:42:09.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 1 -- 172.21.15.37:0/1890733922 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3850627510 0 0) 0x7f0154003520 con 0x7f01680a9d70 2024-03-29T11:42:09.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 1 -- 172.21.15.37:0/1890733922 >> v1:172.21.15.37:6790/0 conn(0x7f016813f650 legacy=0x7f01680b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 1 -- 172.21.15.37:0/1890733922 >> v1:172.21.15.146:6789/0 conn(0x7f01681457f0 legacy=0x7f016813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:09.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 1 monclient: found mon.noname-a 2024-03-29T11:42:09.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:09.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:42:09.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 1 -- 172.21.15.37:0/1890733922 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0168146340 con 0x7f01680a9d70 2024-03-29T11:42:09.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient: _finish_auth 0 2024-03-29T11:42:09.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:09.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:09.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f016f5c1640 5 monclient: authenticate success, global_id 4473 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f016f5c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 1 -- 172.21.15.37:0/1890733922 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f01600015c0 con 0x7f01680a9d70 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient: dump: 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:09.159 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f016f5c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 1 -- 172.21.15.37:0/1890733922 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0160001810 con 0x7f01680a9d70 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f016f5c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f016f5c1640 4 set_mon_vals no callback set 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 1 -- 172.21.15.37:0/1890733922 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0160001ba0 con 0x7f01680a9d70 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.151+0000 7f015ffff640 10 monclient: dump: 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.160 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:09.161 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:09.161 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:09.161 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:09.161 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:09.161 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.152+0000 7f016f5c1640 10 monclient: shutdown 2024-03-29T11:42:09.161 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.152+0000 7f016f5c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:09.161 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.152+0000 7f016f5c1640 1 -- 172.21.15.37:0/1890733922 >> v1:172.21.15.37:6789/0 conn(0x7f01680a9d70 legacy=0x7f016814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.161 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.152+0000 7f016f5c1640 1 -- 172.21.15.37:0/1890733922 shutdown_connections 2024-03-29T11:42:09.161 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.152+0000 7f016f5c1640 1 -- 172.21.15.37:0/1890733922 >> 172.21.15.37:0/1890733922 conn(0x7f0168066a00 msgr2=0x7f0168066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:09.161 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.152+0000 7f016f5c1640 1 -- 172.21.15.37:0/1890733922 shutdown_connections 2024-03-29T11:42:09.163 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.153+0000 7f016f5c1640 1 -- 172.21.15.37:0/1890733922 wait complete. 2024-03-29T11:42:09.163 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.154+0000 7f016f5c1640 10 monclient: build_initial_monmap 2024-03-29T11:42:09.163 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.154+0000 7f016f5c1640 10 monclient: monmap: 2024-03-29T11:42:09.163 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.155+0000 7f016f5c1640 1 librados: starting msgr at 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.155+0000 7f016f5c1640 1 librados: starting objecter 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.155+0000 7f016f5c1640 1 Processor -- start 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.156+0000 7f016f5c1640 1 -- start start 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.156+0000 7f016f5c1640 1 librados: setting wanted keys 2024-03-29T11:42:09.164 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.156+0000 7f016f5c1640 1 librados: calling monclient init 2024-03-29T11:42:09.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.156+0000 7f016f5c1640 10 monclient: init 2024-03-29T11:42:09.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.157+0000 7f016f5c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:09.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.157+0000 7f016f5c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:09.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.157+0000 7f016f5c1640 10 monclient(hunting): picked mon.noname-b con 0x7f01680a9d70 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:09.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.157+0000 7f016f5c1640 10 monclient(hunting): picked mon.noname-c con 0x7f016813f650 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:09.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.157+0000 7f016f5c1640 10 monclient(hunting): picked mon.noname-a con 0x7f01681457f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:09.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.157+0000 7f016f5c1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0168143580 con 0x7f01681457f0 2024-03-29T11:42:09.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.157+0000 7f016f5c1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0168144eb0 con 0x7f01680a9d70 2024-03-29T11:42:09.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.157+0000 7f016f5c1640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016812e8d0 con 0x7f016813f650 2024-03-29T11:42:09.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.157+0000 7f016f5c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:09.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.157+0000 7f016f5c1640 10 monclient(hunting): authenticate will time out at 1308.606661s 2024-03-29T11:42:09.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.162+0000 7f016cb5c640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f016813f650 0x7f016812f2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:42065/0 (socket says 172.21.15.37:42065) 2024-03-29T11:42:09.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.162+0000 7f016cb5c640 1 -- 172.21.15.37:0/3879811845 learned_addr learned my addr 172.21.15.37:0/3879811845 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:09.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.163+0000 7f015dffb640 10 client.?.objecter ms_handle_connect 0x7f016813f650 2024-03-29T11:42:09.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.163+0000 7f015dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:09.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.163+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663420245 0 0) 0x7f016812e8d0 con 0x7f016813f650 2024-03-29T11:42:09.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.163+0000 7f015dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:09.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.163+0000 7f015dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:09.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.163+0000 7f015dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:09.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.163+0000 7f015dffb640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:42:09.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.163+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f014c004eb0 con 0x7f016813f650 2024-03-29T11:42:09.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.163+0000 7f015dffb640 10 client.?.objecter ms_handle_connect 0x7f01680a9d70 2024-03-29T11:42:09.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.163+0000 7f015dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:09.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.164+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2106855712 0 0) 0x7f014c004eb0 con 0x7f016813f650 2024-03-29T11:42:09.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.164+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f016812e8d0 con 0x7f016813f650 2024-03-29T11:42:09.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.164+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103179072 0 0) 0x7f0168144eb0 con 0x7f01680a9d70 2024-03-29T11:42:09.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.164+0000 7f015dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:09.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.164+0000 7f015dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:09.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.164+0000 7f015dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:09.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.164+0000 7f015dffb640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:42:09.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.164+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f014c004eb0 con 0x7f01680a9d70 2024-03-29T11:42:09.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0164001d90 con 0x7f016813f650 2024-03-29T11:42:09.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:09.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 10 monclient(hunting): dump: 2024-03-29T11:42:09.172 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4261152383 0 0) 0x7f014c004eb0 con 0x7f01680a9d70 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0168144eb0 con 0x7f01680a9d70 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2555498322 0 0) 0x7f016812e8d0 con 0x7f016813f650 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 >> v1:172.21.15.146:6789/0 conn(0x7f01680a9d70 legacy=0x7f016812eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 >> v1:172.21.15.37:6789/0 conn(0x7f01681457f0 legacy=0x7f01681ffe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:09.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 1 monclient: found mon.c 2024-03-29T11:42:09.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:09.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:09.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0168143060 con 0x7f016813f650 2024-03-29T11:42:09.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:09.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:09.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.167+0000 7f015dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:09.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.168+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0164001fb0 con 0x7f016813f650 2024-03-29T11:42:09.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.168+0000 7f015dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:09.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.169+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0164003de0 con 0x7f016813f650 2024-03-29T11:42:09.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.169+0000 7f015dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.169+0000 7f015dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:09.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.169+0000 7f015dffb640 10 monclient: dump: 2024-03-29T11:42:09.175 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:09.175 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:09.175 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:09.175 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:09.175 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.175 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:09.175 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:09.175 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:09.175 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:09.175 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:09.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.169+0000 7f016f5c1640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:42:09.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.169+0000 7f016f5c1640 10 monclient: _renew_subs 2024-03-29T11:42:09.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.169+0000 7f016f5c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:09.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.169+0000 7f016f5c1640 1 -- 172.21.15.37:0/3879811845 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016813e690 con 0x7f016813f650 2024-03-29T11:42:09.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.170+0000 7f016f5c1640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:09.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.170+0000 7f016f5c1640 10 monclient: _renew_subs 2024-03-29T11:42:09.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.170+0000 7f016f5c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:09.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.170+0000 7f016f5c1640 1 -- 172.21.15.37:0/3879811845 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01681d6d70 con 0x7f016813f650 2024-03-29T11:42:09.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.170+0000 7f016f5c1640 1 librados: init done 2024-03-29T11:42:09.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.170+0000 7f015effd640 4 set_mon_vals no callback set 2024-03-29T11:42:09.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.173+0000 7f016f5c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:09.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.173+0000 7f016f5c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:09.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.173+0000 7f016f5c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:09.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.173+0000 7f016f5c1640 1 -- 172.21.15.37:0/3879811845 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0148005080 con 0x7f016813f650 2024-03-29T11:42:09.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.174+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f0164003fc0 con 0x7f016813f650 2024-03-29T11:42:09.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.175+0000 7f016d35d640 10 monclient: get_auth_request con 0x7f014c06dd80 auth_method 0 2024-03-29T11:42:09.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.175+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f0164084fe0 con 0x7f016813f650 2024-03-29T11:42:09.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.175+0000 7f015dffb640 10 client.4526.objecter ms_dispatch 0x7f0168000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:09.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.175+0000 7f015dffb640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:09.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.175+0000 7f015dffb640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:09.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.175+0000 7f015dffb640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-29T11:42:09.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.175+0000 7f015dffb640 10 client.4526.objecter ms_handle_connect 0x7f014c06dd80 2024-03-29T11:42:09.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.182+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01640506f0 con 0x7f016813f650 2024-03-29T11:42:09.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.182+0000 7f015dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:09.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.182+0000 7f015dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:09.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.518+0000 7f016f5c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:09.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.518+0000 7f016f5c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:09.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.518+0000 7f016f5c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:09.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.518+0000 7f016f5c1640 1 -- 172.21.15.37:0/3879811845 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0148005330 con 0x7f016813f650 2024-03-29T11:42:09.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.519+0000 7f015dffb640 1 -- 172.21.15.37:0/3879811845 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f01640527f0 con 0x7f016813f650 2024-03-29T11:42:09.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.519+0000 7f015dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:09.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.519+0000 7f015dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:09.525 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:42:09.525 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:09.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.524+0000 7f016f5c1640 10 librados: watch_flush enter 2024-03-29T11:42:09.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.524+0000 7f016f5c1640 10 librados: watch_flush exit 2024-03-29T11:42:09.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.524+0000 7f016f5c1640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-29T11:42:09.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.524+0000 7f016f5c1640 10 client.4526.objecter successfully canceled tick 2024-03-29T11:42:09.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.524+0000 7f016f5c1640 1 -- 172.21.15.37:0/3879811845 >> v1:172.21.15.146:6816/2899149139 conn(0x7f014c06dd80 legacy=0x7f014c070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.524+0000 7f016f5c1640 10 monclient: shutdown 2024-03-29T11:42:09.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.524+0000 7f016f5c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:09.528 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.524+0000 7f016f5c1640 1 -- 172.21.15.37:0/3879811845 >> v1:172.21.15.37:6790/0 conn(0x7f016813f650 legacy=0x7f016812f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.528 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.525+0000 7f016f5c1640 1 -- 172.21.15.37:0/3879811845 shutdown_connections 2024-03-29T11:42:09.528 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.525+0000 7f016f5c1640 1 -- 172.21.15.37:0/3879811845 >> 172.21.15.37:0/3879811845 conn(0x7f0168066a00 msgr2=0x7f01681daae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:09.528 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.526+0000 7f016f5c1640 1 -- 172.21.15.37:0/3879811845 shutdown_connections 2024-03-29T11:42:09.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.526+0000 7f016f5c1640 1 -- 172.21.15.37:0/3879811845 wait complete. 2024-03-29T11:42:09.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:09.526+0000 7f016f5c1640 1 librados: shutdown 2024-03-29T11:42:09.550 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:16.358 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:16.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.508+0000 7f876f896640 10 monclient: get_monmap_and_config 2024-03-29T11:42:16.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.508+0000 7f876f896640 10 monclient: build_initial_monmap 2024-03-29T11:42:16.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.508+0000 7f876f896640 10 monclient: monmap: 2024-03-29T11:42:16.512 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:16.512 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:16.513 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:42:16.509521+0000 2024-03-29T11:42:16.513 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:42:16.509521+0000 2024-03-29T11:42:16.513 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:16.513 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:16.513 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:16.513 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:42:16.513 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:42:16.513 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:16.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.510+0000 7f876f896640 1 Processor -- start 2024-03-29T11:42:16.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.510+0000 7f876f896640 1 -- start start 2024-03-29T11:42:16.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.510+0000 7f876f896640 10 monclient: init 2024-03-29T11:42:16.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.511+0000 7f876f896640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:16.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.511+0000 7f876f896640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:16.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.511+0000 7f876f896640 10 monclient(hunting): picked mon.noname-a con 0x7f87680a9930 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:16.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.511+0000 7f876f896640 10 monclient(hunting): picked mon.noname-b con 0x7f8768149dc0 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:16.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.511+0000 7f876f896640 10 monclient(hunting): picked mon.noname-c con 0x7f876814ff40 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:16.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.511+0000 7f876f896640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87680aae60 con 0x7f87680a9930 2024-03-29T11:42:16.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.511+0000 7f876f896640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87680b4310 con 0x7f876814ff40 2024-03-29T11:42:16.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.511+0000 7f876f896640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87680afdd0 con 0x7f8768149dc0 2024-03-29T11:42:16.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.511+0000 7f876f896640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:16.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.511+0000 7f876f896640 10 monclient(hunting): authenticate will time out at 1315.960710s 2024-03-29T11:42:16.518 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.515+0000 7f876ce31640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f8768149dc0 0x7f876814a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:36361/0 (socket says 172.21.15.37:36361) 2024-03-29T11:42:16.518 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.515+0000 7f876ce31640 1 -- 172.21.15.37:0/1487847277 learned_addr learned my addr 172.21.15.37:0/1487847277 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:16.518 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.516+0000 7f875ffff640 1 -- 172.21.15.37:0/1487847277 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704414903 0 0) 0x7f87680afdd0 con 0x7f8768149dc0 2024-03-29T11:42:16.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.516+0000 7f875ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:16.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.516+0000 7f875ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:16.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.516+0000 7f875ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:16.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.516+0000 7f875ffff640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:42:16.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.516+0000 7f875ffff640 1 -- 172.21.15.37:0/1487847277 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8754003520 con 0x7f8768149dc0 2024-03-29T11:42:16.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 1 -- 172.21.15.37:0/1487847277 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3471989013 0 0) 0x7f8754003520 con 0x7f8768149dc0 2024-03-29T11:42:16.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 1 -- 172.21.15.37:0/1487847277 >> v1:172.21.15.146:6789/0 conn(0x7f876814ff40 legacy=0x7f8768152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 1 -- 172.21.15.37:0/1487847277 >> v1:172.21.15.37:6789/0 conn(0x7f87680a9930 legacy=0x7f8768143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:16.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:16.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 1 monclient: found mon.noname-b 2024-03-29T11:42:16.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:16.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:42:16.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 1 -- 172.21.15.37:0/1487847277 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876814a950 con 0x7f8768149dc0 2024-03-29T11:42:16.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 10 monclient: _finish_auth 0 2024-03-29T11:42:16.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:16.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:16.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 1 -- 172.21.15.37:0/1487847277 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8758001690 con 0x7f8768149dc0 2024-03-29T11:42:16.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:16.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:16.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f875ffff640 10 monclient: dump: 2024-03-29T11:42:16.523 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:16.523 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:16.523 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:16.523 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:16.523 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:16.523 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:16.523 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:16.523 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:16.523 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:16.523 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:16.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f876f896640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:42:16.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.519+0000 7f876f896640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:16.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.520+0000 7f875ffff640 1 -- 172.21.15.37:0/1487847277 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8758001860 con 0x7f8768149dc0 2024-03-29T11:42:16.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.520+0000 7f875ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:16.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.520+0000 7f876f896640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:16.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.520+0000 7f876f896640 4 set_mon_vals no callback set 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.520+0000 7f875ffff640 1 -- 172.21.15.37:0/1487847277 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8758001c70 con 0x7f8768149dc0 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.520+0000 7f875ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.520+0000 7f875ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.520+0000 7f875ffff640 10 monclient: dump: 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:16.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.520+0000 7f876f896640 10 monclient: shutdown 2024-03-29T11:42:16.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.520+0000 7f876f896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:16.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.520+0000 7f876f896640 1 -- 172.21.15.37:0/1487847277 >> v1:172.21.15.37:6790/0 conn(0x7f8768149dc0 legacy=0x7f876814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.521+0000 7f876f896640 1 -- 172.21.15.37:0/1487847277 shutdown_connections 2024-03-29T11:42:16.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.521+0000 7f876f896640 1 -- 172.21.15.37:0/1487847277 >> 172.21.15.37:0/1487847277 conn(0x7f87680b35e0 msgr2=0x7f87680b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:16.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.524+0000 7f876f896640 1 -- 172.21.15.37:0/1487847277 shutdown_connections 2024-03-29T11:42:16.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.524+0000 7f876f896640 1 -- 172.21.15.37:0/1487847277 wait complete. 2024-03-29T11:42:16.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.525+0000 7f876f896640 10 monclient: build_initial_monmap 2024-03-29T11:42:16.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.525+0000 7f876f896640 10 monclient: monmap: 2024-03-29T11:42:16.529 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:16.529 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:16.529 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:42:16.529 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:42:16.529 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:16.529 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:16.529 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:16.529 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:42:16.529 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:42:16.529 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:16.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.526+0000 7f876f896640 1 librados: starting msgr at 2024-03-29T11:42:16.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.526+0000 7f876f896640 1 librados: starting objecter 2024-03-29T11:42:16.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.526+0000 7f876f896640 1 Processor -- start 2024-03-29T11:42:16.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.526+0000 7f876f896640 1 -- start start 2024-03-29T11:42:16.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.526+0000 7f876f896640 1 librados: setting wanted keys 2024-03-29T11:42:16.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.526+0000 7f876f896640 1 librados: calling monclient init 2024-03-29T11:42:16.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.526+0000 7f876f896640 10 monclient: init 2024-03-29T11:42:16.531 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.527+0000 7f876f896640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:16.531 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.527+0000 7f876f896640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:16.531 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.527+0000 7f876f896640 10 monclient(hunting): picked mon.noname-c con 0x7f87680a9930 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:16.531 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.527+0000 7f876f896640 10 monclient(hunting): picked mon.noname-a con 0x7f8768149dc0 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:16.531 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.527+0000 7f876f896640 10 monclient(hunting): picked mon.noname-b con 0x7f876814ff40 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:16.531 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.527+0000 7f876f896640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876806f980 con 0x7f8768149dc0 2024-03-29T11:42:16.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.527+0000 7f876f896640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8768072650 con 0x7f876814ff40 2024-03-29T11:42:16.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.527+0000 7f876f896640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87680714d0 con 0x7f87680a9930 2024-03-29T11:42:16.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.527+0000 7f876f896640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:16.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.527+0000 7f876f896640 10 monclient(hunting): authenticate will time out at 1315.976093s 2024-03-29T11:42:16.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.533+0000 7f876d632640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f87680a9930 0x7f8768071f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:36473/0 (socket says 172.21.15.37:36473) 2024-03-29T11:42:16.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.533+0000 7f876d632640 1 -- 172.21.15.37:0/2428452098 learned_addr learned my addr 172.21.15.37:0/2428452098 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:16.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.533+0000 7f875dffb640 10 client.?.objecter ms_handle_connect 0x7f87680a9930 2024-03-29T11:42:16.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.533+0000 7f875dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:16.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.533+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252489000 0 0) 0x7f87680714d0 con 0x7f87680a9930 2024-03-29T11:42:16.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.533+0000 7f875dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:16.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.533+0000 7f875dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:16.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.533+0000 7f875dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:16.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.533+0000 7f875dffb640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:42:16.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.533+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f874c004eb0 con 0x7f87680a9930 2024-03-29T11:42:16.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3170324814 0 0) 0x7f874c004eb0 con 0x7f87680a9930 2024-03-29T11:42:16.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87680714d0 con 0x7f87680a9930 2024-03-29T11:42:16.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8760004400 con 0x7f87680a9930 2024-03-29T11:42:16.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:16.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:16.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 10 monclient(hunting): dump: 2024-03-29T11:42:16.540 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 10 client.?.objecter ms_handle_connect 0x7f876814ff40 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 10 client.?.objecter ms_handle_connect 0x7f8768149dc0 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:16.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 397944480 0 0) 0x7f87680714d0 con 0x7f87680a9930 2024-03-29T11:42:16.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 >> v1:172.21.15.146:6789/0 conn(0x7f876814ff40 legacy=0x7f876813f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 >> v1:172.21.15.37:6789/0 conn(0x7f8768149dc0 legacy=0x7f876813e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:16.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 1 monclient: found mon.c 2024-03-29T11:42:16.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:16.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:16.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876813ebd0 con 0x7f87680a9930 2024-03-29T11:42:16.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:16.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:16.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f875dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:16.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f876f896640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:42:16.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f876f896640 10 monclient: _renew_subs 2024-03-29T11:42:16.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f876f896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:16.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f876f896640 1 -- 172.21.15.37:0/2428452098 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8768070370 con 0x7f87680a9930 2024-03-29T11:42:16.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f876f896640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:16.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f876f896640 10 monclient: _renew_subs 2024-03-29T11:42:16.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f876f896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:16.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f876f896640 1 -- 172.21.15.37:0/2428452098 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f876806fff0 con 0x7f87680a9930 2024-03-29T11:42:16.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.534+0000 7f876f896640 1 librados: init done 2024-03-29T11:42:16.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.536+0000 7f876f896640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:16.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.537+0000 7f876f896640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:16.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.537+0000 7f876f896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:16.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.537+0000 7f876f896640 1 -- 172.21.15.37:0/2428452098 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87680706c0 con 0x7f87680a9930 2024-03-29T11:42:16.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.541+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8760004720 con 0x7f87680a9930 2024-03-29T11:42:16.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.541+0000 7f875dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:16.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.541+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8760001500 con 0x7f87680a9930 2024-03-29T11:42:16.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.541+0000 7f875dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:16.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.541+0000 7f875dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:16.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.541+0000 7f875dffb640 10 monclient: dump: 2024-03-29T11:42:16.544 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:16.544 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:16.544 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:16.544 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:16.545 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:16.545 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:16.545 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:16.545 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:16.545 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:16.545 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:16.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.541+0000 7f875f7fe640 4 set_mon_vals no callback set 2024-03-29T11:42:16.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.545+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f8760001720 con 0x7f87680a9930 2024-03-29T11:42:16.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.546+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f8760084c80 con 0x7f87680a9930 2024-03-29T11:42:16.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.546+0000 7f875dffb640 10 client.4532.objecter ms_dispatch 0x7f8768000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:16.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.546+0000 7f875dffb640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:16.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.546+0000 7f875dffb640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:16.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.546+0000 7f875dffb640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-29T11:42:16.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.546+0000 7f876ce31640 10 monclient: get_auth_request con 0x7f874c06dd90 auth_method 0 2024-03-29T11:42:16.550 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.547+0000 7f875dffb640 10 client.4532.objecter ms_handle_connect 0x7f874c06dd90 2024-03-29T11:42:16.550 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.548+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8760084e60 con 0x7f87680a9930 2024-03-29T11:42:16.550 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.548+0000 7f875dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:16.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.548+0000 7f875dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:16.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.878+0000 7f876f896640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:16.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.878+0000 7f876f896640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:16.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.878+0000 7f876f896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:16.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.878+0000 7f876f896640 1 -- 172.21.15.37:0/2428452098 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f876806f750 con 0x7f87680a9930 2024-03-29T11:42:16.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.879+0000 7f875dffb640 1 -- 172.21.15.37:0/2428452098 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f876806f750 con 0x7f87680a9930 2024-03-29T11:42:16.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.879+0000 7f875dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:16.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.879+0000 7f875dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:16.883 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:42:16.883 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:16.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.883+0000 7f876f896640 10 librados: watch_flush enter 2024-03-29T11:42:16.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.883+0000 7f876f896640 10 librados: watch_flush exit 2024-03-29T11:42:16.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.883+0000 7f876f896640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-29T11:42:16.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.883+0000 7f876f896640 10 client.4532.objecter successfully canceled tick 2024-03-29T11:42:16.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.885+0000 7f876f896640 1 -- 172.21.15.37:0/2428452098 >> v1:172.21.15.146:6816/2899149139 conn(0x7f874c06dd90 legacy=0x7f874c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.885+0000 7f876f896640 10 monclient: shutdown 2024-03-29T11:42:16.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.885+0000 7f876f896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:16.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.885+0000 7f876f896640 1 -- 172.21.15.37:0/2428452098 >> v1:172.21.15.37:6790/0 conn(0x7f87680a9930 legacy=0x7f8768071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.885+0000 7f876f896640 1 -- 172.21.15.37:0/2428452098 shutdown_connections 2024-03-29T11:42:16.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.885+0000 7f876f896640 1 -- 172.21.15.37:0/2428452098 >> 172.21.15.37:0/2428452098 conn(0x7f87680b35e0 msgr2=0x7f87680b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:16.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.885+0000 7f876f896640 1 -- 172.21.15.37:0/2428452098 shutdown_connections 2024-03-29T11:42:16.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.885+0000 7f876f896640 1 -- 172.21.15.37:0/2428452098 wait complete. 2024-03-29T11:42:16.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:16.885+0000 7f876f896640 1 librados: shutdown 2024-03-29T11:42:16.905 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:23.713 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:23.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.878+0000 7fc70900f640 10 monclient: get_monmap_and_config 2024-03-29T11:42:23.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.878+0000 7fc70900f640 10 monclient: build_initial_monmap 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.878+0000 7fc70900f640 10 monclient: monmap: 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:42:23.878936+0000 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:42:23.878936+0000 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.879+0000 7fc70900f640 1 Processor -- start 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.879+0000 7fc70900f640 1 -- start start 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.879+0000 7fc70900f640 10 monclient: init 2024-03-29T11:42:23.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.880+0000 7fc70900f640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:23.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.880+0000 7fc70900f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:23.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.880+0000 7fc70900f640 10 monclient(hunting): picked mon.noname-b con 0x7fc7040af6c0 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:23.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.880+0000 7fc70900f640 10 monclient(hunting): picked mon.noname-c con 0x7fc704153210 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:23.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.883+0000 7fc70900f640 10 monclient(hunting): picked mon.noname-a con 0x7fc70414fdc0 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:23.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.883+0000 7fc70900f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7040a94e0 con 0x7fc70414fdc0 2024-03-29T11:42:23.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.883+0000 7fc70900f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7040af3a0 con 0x7fc704153210 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.883+0000 7fc70900f640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7040afb60 con 0x7fc7040af6c0 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.883+0000 7fc70900f640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.883+0000 7fc70900f640 10 monclient(hunting): authenticate will time out at 1323.332472s 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.886+0000 7fc70259c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc704153210 0x7fc7041578c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:52513/0 (socket says 172.21.15.37:52513) 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.886+0000 7fc70259c640 1 -- 172.21.15.37:0/692649056 learned_addr learned my addr 172.21.15.37:0/692649056 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.887+0000 7fc701d9b640 1 -- 172.21.15.37:0/692649056 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986194335 0 0) 0x7fc7040a94e0 con 0x7fc70414fdc0 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.887+0000 7fc701d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.887+0000 7fc701d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.887+0000 7fc701d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.887+0000 7fc701d9b640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.887+0000 7fc701d9b640 1 -- 172.21.15.37:0/692649056 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6ec003520 con 0x7fc70414fdc0 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 1 -- 172.21.15.37:0/692649056 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002112002 0 0) 0x7fc7040af3a0 con 0x7fc704153210 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:23.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 1 -- 172.21.15.37:0/692649056 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7040a94e0 con 0x7fc704153210 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 1 -- 172.21.15.37:0/692649056 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1009443035 0 0) 0x7fc6ec003520 con 0x7fc70414fdc0 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 1 -- 172.21.15.37:0/692649056 >> v1:172.21.15.37:6790/0 conn(0x7fc7040af6c0 legacy=0x7fc70414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 1 -- 172.21.15.37:0/692649056 >> v1:172.21.15.146:6789/0 conn(0x7fc704153210 legacy=0x7fc7041578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 1 monclient: found mon.noname-a 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 1 -- 172.21.15.37:0/692649056 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7040aab80 con 0x7fc70414fdc0 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient: _finish_auth 0 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 1 -- 172.21.15.37:0/692649056 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc6fc001940 con 0x7fc70414fdc0 2024-03-29T11:42:23.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc70900f640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc70900f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient: dump: 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:23.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc70900f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:23.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 1 -- 172.21.15.37:0/692649056 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6fc001b60 con 0x7fc70414fdc0 2024-03-29T11:42:23.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:23.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc70900f640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:23.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc70900f640 4 set_mon_vals no callback set 2024-03-29T11:42:23.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 1 -- 172.21.15.37:0/692649056 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc6fc001ef0 con 0x7fc70414fdc0 2024-03-29T11:42:23.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:23.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:23.895 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.888+0000 7fc701d9b640 10 monclient: dump: 2024-03-29T11:42:23.895 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:23.895 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:23.895 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:23.895 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:23.895 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:23.895 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:23.896 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:23.896 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:23.896 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:23.896 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:23.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.889+0000 7fc70900f640 10 monclient: shutdown 2024-03-29T11:42:23.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.889+0000 7fc70900f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:23.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.889+0000 7fc70900f640 1 -- 172.21.15.37:0/692649056 >> v1:172.21.15.37:6789/0 conn(0x7fc70414fdc0 legacy=0x7fc704152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:23.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.889+0000 7fc70900f640 1 -- 172.21.15.37:0/692649056 shutdown_connections 2024-03-29T11:42:23.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.889+0000 7fc70900f640 1 -- 172.21.15.37:0/692649056 >> 172.21.15.37:0/692649056 conn(0x7fc7040b35e0 msgr2=0x7fc7040b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.889+0000 7fc70900f640 1 -- 172.21.15.37:0/692649056 shutdown_connections 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.889+0000 7fc70900f640 1 -- 172.21.15.37:0/692649056 wait complete. 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.891+0000 7fc70900f640 10 monclient: build_initial_monmap 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.891+0000 7fc70900f640 10 monclient: monmap: 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:23.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.891+0000 7fc70900f640 1 librados: starting msgr at 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.891+0000 7fc70900f640 1 librados: starting objecter 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.891+0000 7fc70900f640 1 Processor -- start 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.891+0000 7fc70900f640 1 -- start start 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.891+0000 7fc70900f640 1 librados: setting wanted keys 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.891+0000 7fc70900f640 1 librados: calling monclient init 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.891+0000 7fc70900f640 10 monclient: init 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.892+0000 7fc70900f640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.892+0000 7fc70900f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.892+0000 7fc70900f640 10 monclient(hunting): picked mon.noname-c con 0x7fc7040af6c0 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.892+0000 7fc70900f640 10 monclient(hunting): picked mon.noname-b con 0x7fc70414fdc0 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.892+0000 7fc70900f640 10 monclient(hunting): picked mon.noname-a con 0x7fc704153210 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.892+0000 7fc70900f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc704154d60 con 0x7fc704153210 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.892+0000 7fc70900f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc704155040 con 0x7fc70414fdc0 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.892+0000 7fc70900f640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc704153ba0 con 0x7fc7040af6c0 2024-03-29T11:42:23.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.892+0000 7fc70900f640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:23.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.892+0000 7fc70900f640 10 monclient(hunting): authenticate will time out at 1323.341593s 2024-03-29T11:42:23.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.896+0000 7fc702d9d640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fc7040af6c0 0x7fc704071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:40839/0 (socket says 172.21.15.37:40839) 2024-03-29T11:42:23.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.896+0000 7fc702d9d640 1 -- 172.21.15.37:0/3373508673 learned_addr learned my addr 172.21.15.37:0/3373508673 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:23.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.897+0000 7fc6eb7fe640 10 client.?.objecter ms_handle_connect 0x7fc7040af6c0 2024-03-29T11:42:23.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.897+0000 7fc6eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:23.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.899+0000 7fc6eb7fe640 10 client.?.objecter ms_handle_connect 0x7fc70414fdc0 2024-03-29T11:42:23.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.899+0000 7fc6eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:23.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.899+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785047030 0 0) 0x7fc704155040 con 0x7fc70414fdc0 2024-03-29T11:42:23.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.899+0000 7fc6eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:23.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.899+0000 7fc6eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:23.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.899+0000 7fc6eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:23.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.899+0000 7fc6eb7fe640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:42:23.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.899+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f0004eb0 con 0x7fc70414fdc0 2024-03-29T11:42:23.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.899+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281591811 0 0) 0x7fc704153ba0 con 0x7fc7040af6c0 2024-03-29T11:42:23.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.899+0000 7fc6eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:23.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.899+0000 7fc6eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:23.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.899+0000 7fc6eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:23.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.899+0000 7fc6eb7fe640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:42:23.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.899+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc704155040 con 0x7fc7040af6c0 2024-03-29T11:42:23.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 10 client.?.objecter ms_handle_connect 0x7fc704153210 2024-03-29T11:42:23.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:23.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1747691843 0 0) 0x7fc6f0004eb0 con 0x7fc70414fdc0 2024-03-29T11:42:23.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc704153ba0 con 0x7fc70414fdc0 2024-03-29T11:42:23.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc6f4001850 con 0x7fc70414fdc0 2024-03-29T11:42:23.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:23.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:23.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 10 monclient(hunting): dump: 2024-03-29T11:42:23.905 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:23.905 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:23.905 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:23.905 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:23.905 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:23.905 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:23.905 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:23.905 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832730977 0 0) 0x7fc704154d60 con 0x7fc704153210 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.900+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f0004eb0 con 0x7fc704153210 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4096920414 0 0) 0x7fc704153ba0 con 0x7fc70414fdc0 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 >> v1:172.21.15.37:6790/0 conn(0x7fc7040af6c0 legacy=0x7fc704071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 >> v1:172.21.15.37:6789/0 conn(0x7fc704153210 legacy=0x7fc7042114a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc6eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc6eb7fe640 1 monclient: found mon.b 2024-03-29T11:42:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc6eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc6eb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc704153d30 con 0x7fc70414fdc0 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc6eb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc6eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc6eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc70900f640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc70900f640 10 monclient: _renew_subs 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc70900f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc70900f640 1 -- 172.21.15.37:0/3373508673 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc704072510 con 0x7fc70414fdc0 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc70900f640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc70900f640 10 monclient: _renew_subs 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc70900f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc70900f640 1 -- 172.21.15.37:0/3373508673 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70414df60 con 0x7fc70414fdc0 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.901+0000 7fc70900f640 1 librados: init done 2024-03-29T11:42:23.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.902+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6f4001d60 con 0x7fc70414fdc0 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.902+0000 7fc6eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.902+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc6f4004980 con 0x7fc70414fdc0 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.902+0000 7fc6eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.902+0000 7fc6eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.902+0000 7fc6eb7fe640 10 monclient: dump: 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:23.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.902+0000 7fc70159a640 4 set_mon_vals no callback set 2024-03-29T11:42:23.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.902+0000 7fc70900f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:23.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.904+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fc6f4004ae0 con 0x7fc70414fdc0 2024-03-29T11:42:23.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.904+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fc6f40849a0 con 0x7fc70414fdc0 2024-03-29T11:42:23.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.904+0000 7fc6eb7fe640 10 client.4447.objecter ms_dispatch 0x7fc704000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:23.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.904+0000 7fc6eb7fe640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:23.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.904+0000 7fc6eb7fe640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:23.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.905+0000 7fc6eb7fe640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-29T11:42:23.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.905+0000 7fc70900f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:23.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.905+0000 7fc70900f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:42:23.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.905+0000 7fc70900f640 1 -- 172.21.15.37:0/3373508673 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e4005080 con 0x7fc70414fdc0 2024-03-29T11:42:23.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.905+0000 7fc702d9d640 10 monclient: get_auth_request con 0x7fc6f006df20 auth_method 0 2024-03-29T11:42:23.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.906+0000 7fc6eb7fe640 10 client.4447.objecter ms_handle_connect 0x7fc6f006df20 2024-03-29T11:42:23.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.911+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6f4050280 con 0x7fc70414fdc0 2024-03-29T11:42:23.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.912+0000 7fc6eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:23.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:23.912+0000 7fc6eb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:24.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.258+0000 7fc70900f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:24.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.258+0000 7fc70900f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:24.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.258+0000 7fc70900f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:42:24.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.258+0000 7fc70900f640 1 -- 172.21.15.37:0/3373508673 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6e4005330 con 0x7fc70414fdc0 2024-03-29T11:42:24.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.259+0000 7fc6eb7fe640 1 -- 172.21.15.37:0/3373508673 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fc6f40521b0 con 0x7fc70414fdc0 2024-03-29T11:42:24.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.259+0000 7fc6eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:24.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.259+0000 7fc6eb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:24.263 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:42:24.263 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:24.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.264+0000 7fc70900f640 10 librados: watch_flush enter 2024-03-29T11:42:24.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.264+0000 7fc70900f640 10 librados: watch_flush exit 2024-03-29T11:42:24.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.264+0000 7fc70900f640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-29T11:42:24.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.264+0000 7fc70900f640 10 client.4447.objecter successfully canceled tick 2024-03-29T11:42:24.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.264+0000 7fc70900f640 1 -- 172.21.15.37:0/3373508673 >> v1:172.21.15.146:6816/2899149139 conn(0x7fc6f006df20 legacy=0x7fc6f00703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:24.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.264+0000 7fc70900f640 10 monclient: shutdown 2024-03-29T11:42:24.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.264+0000 7fc70900f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:24.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.264+0000 7fc70900f640 1 -- 172.21.15.37:0/3373508673 >> v1:172.21.15.146:6789/0 conn(0x7fc70414fdc0 legacy=0x7fc704148260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:24.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.265+0000 7fc70900f640 1 -- 172.21.15.37:0/3373508673 shutdown_connections 2024-03-29T11:42:24.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.265+0000 7fc70900f640 1 -- 172.21.15.37:0/3373508673 >> 172.21.15.37:0/3373508673 conn(0x7fc7040b35e0 msgr2=0x7fc7040ac140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:24.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.266+0000 7fc70900f640 1 -- 172.21.15.37:0/3373508673 shutdown_connections 2024-03-29T11:42:24.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.266+0000 7fc70900f640 1 -- 172.21.15.37:0/3373508673 wait complete. 2024-03-29T11:42:24.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:24.266+0000 7fc70900f640 1 librados: shutdown 2024-03-29T11:42:24.286 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:31.095 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:31.253 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.249+0000 7f468fb8d640 10 monclient: get_monmap_and_config 2024-03-29T11:42:31.254 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.249+0000 7f468fb8d640 10 monclient: build_initial_monmap 2024-03-29T11:42:31.254 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.249+0000 7f468fb8d640 10 monclient: monmap: 2024-03-29T11:42:31.254 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:31.254 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:31.254 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:42:31.250617+0000 2024-03-29T11:42:31.254 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:42:31.250617+0000 2024-03-29T11:42:31.254 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:31.254 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:31.254 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:31.254 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:42:31.254 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:42:31.254 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:31.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.252+0000 7f468fb8d640 1 Processor -- start 2024-03-29T11:42:31.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.253+0000 7f468fb8d640 1 -- start start 2024-03-29T11:42:31.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.253+0000 7f468fb8d640 10 monclient: init 2024-03-29T11:42:31.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.254+0000 7f468fb8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:31.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.254+0000 7f468fb8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:31.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.254+0000 7f468fb8d640 10 monclient(hunting): picked mon.noname-a con 0x7f46880a9ac0 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:31.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.254+0000 7f468fb8d640 10 monclient(hunting): picked mon.noname-c con 0x7f468813ee90 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:31.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.254+0000 7f468fb8d640 10 monclient(hunting): picked mon.noname-b con 0x7f46880af900 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:31.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.254+0000 7f468fb8d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46880afd10 con 0x7f46880a9ac0 2024-03-29T11:42:31.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.254+0000 7f468fb8d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46880af6f0 con 0x7f468813ee90 2024-03-29T11:42:31.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.254+0000 7f468fb8d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46880aac80 con 0x7f46880af900 2024-03-29T11:42:31.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.254+0000 7f468fb8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:31.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.254+0000 7f468fb8d640 10 monclient(hunting): authenticate will time out at 1330.703232s 2024-03-29T11:42:31.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.259+0000 7f468d929640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f46880a9ac0 0x7f46880adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41130/0 (socket says 172.21.15.37:41130) 2024-03-29T11:42:31.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.259+0000 7f468d929640 1 -- 172.21.15.37:0/3774996538 learned_addr learned my addr 172.21.15.37:0/3774996538 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:31.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.261+0000 7f468c927640 1 -- 172.21.15.37:0/3774996538 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452960646 0 0) 0x7f46880af6f0 con 0x7f468813ee90 2024-03-29T11:42:31.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.261+0000 7f468c927640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.261+0000 7f468c927640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.261+0000 7f468c927640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.261+0000 7f468c927640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:42:31.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.261+0000 7f468c927640 1 -- 172.21.15.37:0/3774996538 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4674003520 con 0x7f468813ee90 2024-03-29T11:42:31.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.262+0000 7f468c927640 1 -- 172.21.15.37:0/3774996538 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708003719 0 0) 0x7f46880afd10 con 0x7f46880a9ac0 2024-03-29T11:42:31.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.262+0000 7f468c927640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:42:31.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 1 -- 172.21.15.37:0/3774996538 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46880af6f0 con 0x7f46880a9ac0 2024-03-29T11:42:31.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 1 -- 172.21.15.37:0/3774996538 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2236241106 0 0) 0x7f4674003520 con 0x7f468813ee90 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 1 -- 172.21.15.37:0/3774996538 >> v1:172.21.15.37:6790/0 conn(0x7f46880af900 legacy=0x7f4688149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 1 -- 172.21.15.37:0/3774996538 >> v1:172.21.15.37:6789/0 conn(0x7f46880a9ac0 legacy=0x7f46880adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 10 monclient: _finish_hunting 0 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 1 monclient: found mon.noname-c 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 1 -- 172.21.15.37:0/3774996538 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46880aaf60 con 0x7f468813ee90 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 10 monclient: _finish_auth 0 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 10 monclient: _check_auth_tickets 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 1 -- 172.21.15.37:0/3774996538 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4678001610 con 0x7f468813ee90 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468c927640 10 monclient: dump: 2024-03-29T11:42:31.268 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:31.269 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:31.269 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:31.269 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:31.269 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.269 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:31.269 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:31.269 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:31.269 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:31.269 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:31.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468fb8d640 5 monclient: authenticate success, global_id 4450 2024-03-29T11:42:31.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.263+0000 7f468fb8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:31.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.264+0000 7f468c927640 1 -- 172.21.15.37:0/3774996538 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4678001830 con 0x7f468813ee90 2024-03-29T11:42:31.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.264+0000 7f468c927640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:31.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.264+0000 7f468c927640 1 -- 172.21.15.37:0/3774996538 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4678001bc0 con 0x7f468813ee90 2024-03-29T11:42:31.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.264+0000 7f468c927640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.264+0000 7f468c927640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:31.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.264+0000 7f468c927640 10 monclient: dump: 2024-03-29T11:42:31.270 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.264+0000 7f468fb8d640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.264+0000 7f468fb8d640 4 set_mon_vals no callback set 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.265+0000 7f468fb8d640 10 monclient: shutdown 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.265+0000 7f468fb8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.265+0000 7f468fb8d640 1 -- 172.21.15.37:0/3774996538 >> v1:172.21.15.146:6789/0 conn(0x7f468813ee90 legacy=0x7f468813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.266+0000 7f468fb8d640 1 -- 172.21.15.37:0/3774996538 shutdown_connections 2024-03-29T11:42:31.272 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.266+0000 7f468fb8d640 1 -- 172.21.15.37:0/3774996538 >> 172.21.15.37:0/3774996538 conn(0x7f46880b35e0 msgr2=0x7f46880b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:31.272 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.266+0000 7f468fb8d640 1 -- 172.21.15.37:0/3774996538 shutdown_connections 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.266+0000 7f468fb8d640 1 -- 172.21.15.37:0/3774996538 wait complete. 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.267+0000 7f468fb8d640 10 monclient: build_initial_monmap 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.267+0000 7f468fb8d640 10 monclient: monmap: 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.267+0000 7f468fb8d640 1 librados: starting msgr at 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.267+0000 7f468fb8d640 1 librados: starting objecter 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.267+0000 7f468fb8d640 1 Processor -- start 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.268+0000 7f468fb8d640 1 -- start start 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.268+0000 7f468fb8d640 1 librados: setting wanted keys 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.268+0000 7f468fb8d640 1 librados: calling monclient init 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.268+0000 7f468fb8d640 10 monclient: init 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.268+0000 7f468fb8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.268+0000 7f468fb8d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.268+0000 7f468fb8d640 10 monclient(hunting): picked mon.noname-b con 0x7f46880a9ac0 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.268+0000 7f468fb8d640 10 monclient(hunting): picked mon.noname-c con 0x7f46880af900 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.268+0000 7f468fb8d640 10 monclient(hunting): picked mon.noname-a con 0x7f468813ee90 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.268+0000 7f468fb8d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4688140380 con 0x7f468813ee90 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.268+0000 7f468fb8d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4688142520 con 0x7f46880a9ac0 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.268+0000 7f468fb8d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46881416f0 con 0x7f46880af900 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.268+0000 7f468fb8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.268+0000 7f468fb8d640 10 monclient(hunting): authenticate will time out at 1330.717833s 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.272+0000 7f468e12a640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f468813ee90 0x7f4688071010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41225/0 (socket says 172.21.15.37:41225) 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.272+0000 7f468e12a640 1 -- 172.21.15.37:0/196253674 learned_addr learned my addr 172.21.15.37:0/196253674 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:31.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.273+0000 7f467e7fc640 10 client.?.objecter ms_handle_connect 0x7f468813ee90 2024-03-29T11:42:31.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.273+0000 7f467e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:31.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.273+0000 7f467e7fc640 10 client.?.objecter ms_handle_connect 0x7f46880a9ac0 2024-03-29T11:42:31.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.273+0000 7f467e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:31.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.273+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156435234 0 0) 0x7f4688140380 con 0x7f468813ee90 2024-03-29T11:42:31.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.273+0000 7f467e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.273+0000 7f467e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.273+0000 7f467e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.273+0000 7f467e7fc640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.273+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f466c004eb0 con 0x7f468813ee90 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.274+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519422997 0 0) 0x7f4688142520 con 0x7f46880a9ac0 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.274+0000 7f467e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.274+0000 7f467e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.274+0000 7f467e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.274+0000 7f467e7fc640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.274+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4688140380 con 0x7f46880a9ac0 2024-03-29T11:42:31.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1696586353 0 0) 0x7f4688140380 con 0x7f46880a9ac0 2024-03-29T11:42:31.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4688142520 con 0x7f46880a9ac0 2024-03-29T11:42:31.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 810305742 0 0) 0x7f466c004eb0 con 0x7f468813ee90 2024-03-29T11:42:31.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4688140380 con 0x7f468813ee90 2024-03-29T11:42:31.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f46840018b0 con 0x7f46880a9ac0 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4680001be0 con 0x7f468813ee90 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:31.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1842555776 0 0) 0x7f4688142520 con 0x7f46880a9ac0 2024-03-29T11:42:31.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 >> v1:172.21.15.37:6790/0 conn(0x7f46880af900 legacy=0x7f46881e3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 >> v1:172.21.15.37:6789/0 conn(0x7f468813ee90 legacy=0x7f4688071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.277+0000 7f467e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:31.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f467e7fc640 1 monclient: found mon.b 2024-03-29T11:42:31.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f467e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:31.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f467e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:42:31.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46881e45e0 con 0x7f46880a9ac0 2024-03-29T11:42:31.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f467e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:31.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f467e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:31.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f467e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:31.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f468fb8d640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:42:31.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f468fb8d640 10 monclient: _renew_subs 2024-03-29T11:42:31.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f468fb8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:42:31.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f468fb8d640 1 -- 172.21.15.37:0/196253674 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4688071720 con 0x7f46880a9ac0 2024-03-29T11:42:31.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f468fb8d640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f468fb8d640 10 monclient: _renew_subs 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f468fb8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f468fb8d640 1 -- 172.21.15.37:0/196253674 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4688071d70 con 0x7f46880a9ac0 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f468fb8d640 1 librados: init done 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46840047e0 con 0x7f46880a9ac0 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f467e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4684001d80 con 0x7f46880a9ac0 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f467e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f467e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.278+0000 7f467e7fc640 10 monclient: dump: 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.279+0000 7f468fb8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.280+0000 7f468fb8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.280+0000 7f468fb8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.280+0000 7f468fb8d640 1 -- 172.21.15.37:0/196253674 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46881426b0 con 0x7f46880a9ac0 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.280+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f4684004cb0 con 0x7f46880a9ac0 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.281+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f4684084700 con 0x7f46880a9ac0 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.281+0000 7f467e7fc640 10 client.4453.objecter ms_dispatch 0x7f4688000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.281+0000 7f467e7fc640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.281+0000 7f467e7fc640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.281+0000 7f467e7fc640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.281+0000 7f467ffff640 4 set_mon_vals no callback set 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.282+0000 7f468d128640 10 monclient: get_auth_request con 0x7f466c06e0d0 auth_method 0 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.282+0000 7f467e7fc640 10 client.4453.objecter ms_handle_connect 0x7f466c06e0d0 2024-03-29T11:42:31.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.289+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4684050040 con 0x7f46880a9ac0 2024-03-29T11:42:31.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.289+0000 7f467e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:31.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.289+0000 7f467e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:31.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.654+0000 7f468fb8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:31.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.654+0000 7f468fb8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:31.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.654+0000 7f468fb8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:42:31.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.654+0000 7f468fb8d640 1 -- 172.21.15.37:0/196253674 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4688140150 con 0x7f46880a9ac0 2024-03-29T11:42:31.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.654+0000 7f467e7fc640 1 -- 172.21.15.37:0/196253674 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f4684050040 con 0x7f46880a9ac0 2024-03-29T11:42:31.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.654+0000 7f467e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:31.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.654+0000 7f467e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:31.658 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:42:31.658 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:31.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.658+0000 7f468fb8d640 10 librados: watch_flush enter 2024-03-29T11:42:31.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.658+0000 7f468fb8d640 10 librados: watch_flush exit 2024-03-29T11:42:31.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.658+0000 7f468fb8d640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T11:42:31.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.658+0000 7f468fb8d640 10 client.4453.objecter successfully canceled tick 2024-03-29T11:42:31.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.659+0000 7f468fb8d640 1 -- 172.21.15.37:0/196253674 >> v1:172.21.15.146:6816/2899149139 conn(0x7f466c06e0d0 legacy=0x7f466c070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.659+0000 7f468fb8d640 10 monclient: shutdown 2024-03-29T11:42:31.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.659+0000 7f468fb8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:31.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.659+0000 7f468fb8d640 1 -- 172.21.15.37:0/196253674 >> v1:172.21.15.146:6789/0 conn(0x7f46880a9ac0 legacy=0x7f4688140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.659+0000 7f468fb8d640 1 -- 172.21.15.37:0/196253674 shutdown_connections 2024-03-29T11:42:31.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.659+0000 7f468fb8d640 1 -- 172.21.15.37:0/196253674 >> 172.21.15.37:0/196253674 conn(0x7f46880b35e0 msgr2=0x7f46880adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:31.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.660+0000 7f468fb8d640 1 -- 172.21.15.37:0/196253674 shutdown_connections 2024-03-29T11:42:31.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.660+0000 7f468fb8d640 1 -- 172.21.15.37:0/196253674 wait complete. 2024-03-29T11:42:31.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:31.661+0000 7f468fb8d640 1 librados: shutdown 2024-03-29T11:42:31.685 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:38.493 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:38.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.642+0000 7f959c87b640 10 monclient: get_monmap_and_config 2024-03-29T11:42:38.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.642+0000 7f959c87b640 10 monclient: build_initial_monmap 2024-03-29T11:42:38.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.642+0000 7f959c87b640 10 monclient: monmap: 2024-03-29T11:42:38.646 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:38.646 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:38.646 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:42:38.643571+0000 2024-03-29T11:42:38.646 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:42:38.643571+0000 2024-03-29T11:42:38.646 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:38.646 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:38.646 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:38.646 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:42:38.647 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:42:38.647 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:38.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.643+0000 7f959c87b640 1 Processor -- start 2024-03-29T11:42:38.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.643+0000 7f959c87b640 1 -- start start 2024-03-29T11:42:38.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.643+0000 7f959c87b640 10 monclient: init 2024-03-29T11:42:38.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.645+0000 7f959c87b640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:38.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.645+0000 7f959c87b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:38.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.645+0000 7f959c87b640 10 monclient(hunting): picked mon.noname-c con 0x7f95981457f0 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:38.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.645+0000 7f959c87b640 10 monclient(hunting): picked mon.noname-a con 0x7f959813f650 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:38.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.645+0000 7f959c87b640 10 monclient(hunting): picked mon.noname-b con 0x7f95980a9d70 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:38.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.645+0000 7f959c87b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95980aaef0 con 0x7f959813f650 2024-03-29T11:42:38.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.645+0000 7f959c87b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9598145bd0 con 0x7f95981457f0 2024-03-29T11:42:38.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.645+0000 7f9595d9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f959813f650 0x7f95980b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:60590/0 (socket says 172.21.15.37:60590) 2024-03-29T11:42:38.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.645+0000 7f9595d9b640 1 -- 172.21.15.37:0/1543868386 learned_addr learned my addr 172.21.15.37:0/1543868386 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:38.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.645+0000 7f959c87b640 1 -- 172.21.15.37:0/1543868386 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9598145eb0 con 0x7f95980a9d70 2024-03-29T11:42:38.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.645+0000 7f959c87b640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:38.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.645+0000 7f959c87b640 10 monclient(hunting): authenticate will time out at 1338.094865s 2024-03-29T11:42:38.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 1 -- 172.21.15.37:0/1543868386 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529361807 0 0) 0x7f9598145eb0 con 0x7f95980a9d70 2024-03-29T11:42:38.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:38.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:38.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:38.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:42:38.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 1 -- 172.21.15.37:0/1543868386 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9580003520 con 0x7f95980a9d70 2024-03-29T11:42:38.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 1 -- 172.21.15.37:0/1543868386 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247650745 0 0) 0x7f95980aaef0 con 0x7f959813f650 2024-03-29T11:42:38.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:38.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:38.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:38.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:42:38.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 1 -- 172.21.15.37:0/1543868386 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9598145eb0 con 0x7f959813f650 2024-03-29T11:42:38.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 1 -- 172.21.15.37:0/1543868386 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463847405 0 0) 0x7f9580003520 con 0x7f95980a9d70 2024-03-29T11:42:38.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 1 -- 172.21.15.37:0/1543868386 >> v1:172.21.15.146:6789/0 conn(0x7f95981457f0 legacy=0x7f959813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:38.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 1 -- 172.21.15.37:0/1543868386 >> v1:172.21.15.37:6789/0 conn(0x7f959813f650 legacy=0x7f95980b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:38.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient: _finish_hunting 0 2024-03-29T11:42:38.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 1 monclient: found mon.noname-b 2024-03-29T11:42:38.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:38.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:42:38.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 1 -- 172.21.15.37:0/1543868386 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9598146340 con 0x7f95980a9d70 2024-03-29T11:42:38.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient: _finish_auth 0 2024-03-29T11:42:38.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient: _check_auth_tickets 2024-03-29T11:42:38.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:38.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 1 -- 172.21.15.37:0/1543868386 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9590001a10 con 0x7f95980a9d70 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959559a640 10 monclient: dump: 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959c87b640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:42:38.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.646+0000 7f959c87b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.647+0000 7f959559a640 1 -- 172.21.15.37:0/1543868386 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9590001b70 con 0x7f95980a9d70 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.647+0000 7f959559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.647+0000 7f959559a640 1 -- 172.21.15.37:0/1543868386 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9590001f80 con 0x7f95980a9d70 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.647+0000 7f959559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.647+0000 7f959559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.647+0000 7f959559a640 10 monclient: dump: 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:38.655 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:38.656 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:38.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.647+0000 7f959c87b640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:38.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.647+0000 7f959c87b640 4 set_mon_vals no callback set 2024-03-29T11:42:38.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.648+0000 7f959c87b640 10 monclient: shutdown 2024-03-29T11:42:38.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.648+0000 7f959c87b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:38.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.648+0000 7f959c87b640 1 -- 172.21.15.37:0/1543868386 >> v1:172.21.15.37:6790/0 conn(0x7f95980a9d70 legacy=0x7f959814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:38.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.648+0000 7f959c87b640 1 -- 172.21.15.37:0/1543868386 shutdown_connections 2024-03-29T11:42:38.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.648+0000 7f959c87b640 1 -- 172.21.15.37:0/1543868386 >> 172.21.15.37:0/1543868386 conn(0x7f9598066a00 msgr2=0x7f9598066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:38.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.648+0000 7f959c87b640 1 -- 172.21.15.37:0/1543868386 shutdown_connections 2024-03-29T11:42:38.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.648+0000 7f959c87b640 1 -- 172.21.15.37:0/1543868386 wait complete. 2024-03-29T11:42:38.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.650+0000 7f959c87b640 10 monclient: build_initial_monmap 2024-03-29T11:42:38.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.650+0000 7f959c87b640 10 monclient: monmap: 2024-03-29T11:42:38.666 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:38.666 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:38.666 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:42:38.666 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:42:38.666 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:38.667 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:38.667 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:38.667 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:42:38.667 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:42:38.667 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:38.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.650+0000 7f959c87b640 1 librados: starting msgr at 2024-03-29T11:42:38.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.650+0000 7f959c87b640 1 librados: starting objecter 2024-03-29T11:42:38.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.650+0000 7f959c87b640 1 Processor -- start 2024-03-29T11:42:38.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.650+0000 7f959c87b640 1 -- start start 2024-03-29T11:42:38.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.651+0000 7f959c87b640 1 librados: setting wanted keys 2024-03-29T11:42:38.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.651+0000 7f959c87b640 1 librados: calling monclient init 2024-03-29T11:42:38.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.651+0000 7f959c87b640 10 monclient: init 2024-03-29T11:42:38.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.652+0000 7f959c87b640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:38.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.652+0000 7f959c87b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:38.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.652+0000 7f959c87b640 10 monclient(hunting): picked mon.noname-b con 0x7f95980a9d70 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:38.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.652+0000 7f959c87b640 10 monclient(hunting): picked mon.noname-a con 0x7f959813f650 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:38.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.652+0000 7f959c87b640 10 monclient(hunting): picked mon.noname-c con 0x7f95981457f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:38.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.652+0000 7f959c87b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9598140310 con 0x7f959813f650 2024-03-29T11:42:38.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.652+0000 7f959659c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f95980a9d70 0x7f95981407a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:49288/0 (socket says 172.21.15.37:49288) 2024-03-29T11:42:38.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.652+0000 7f959659c640 1 -- 172.21.15.37:0/2068971563 learned_addr learned my addr 172.21.15.37:0/2068971563 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:38.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.652+0000 7f959c87b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95981ccf20 con 0x7f95980a9d70 2024-03-29T11:42:38.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.652+0000 7f959c87b640 1 -- 172.21.15.37:0/2068971563 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f959813fda0 con 0x7f95981457f0 2024-03-29T11:42:38.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.652+0000 7f959c87b640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:38.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.652+0000 7f959c87b640 10 monclient(hunting): authenticate will time out at 1338.101853s 2024-03-29T11:42:38.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.652+0000 7f9586ffd640 10 client.?.objecter ms_handle_connect 0x7f95980a9d70 2024-03-29T11:42:38.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.653+0000 7f9586ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:38.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.653+0000 7f9586ffd640 10 client.?.objecter ms_handle_connect 0x7f95981457f0 2024-03-29T11:42:38.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.653+0000 7f9586ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:38.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.653+0000 7f9586ffd640 10 client.?.objecter ms_handle_connect 0x7f959813f650 2024-03-29T11:42:38.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.653+0000 7f9586ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:38.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.653+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610885065 0 0) 0x7f959813fda0 con 0x7f95981457f0 2024-03-29T11:42:38.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.653+0000 7f9586ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:38.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.653+0000 7f9586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:38.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.653+0000 7f9586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:38.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.653+0000 7f9586ffd640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:42:38.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.653+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9578004eb0 con 0x7f95981457f0 2024-03-29T11:42:38.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.654+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623627828 0 0) 0x7f9598140310 con 0x7f959813f650 2024-03-29T11:42:38.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.654+0000 7f9586ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:38.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.654+0000 7f9586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:38.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.654+0000 7f9586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:38.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.654+0000 7f9586ffd640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:42:38.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.654+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f959813fda0 con 0x7f959813f650 2024-03-29T11:42:38.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.654+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243022381 0 0) 0x7f95981ccf20 con 0x7f95980a9d70 2024-03-29T11:42:38.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.654+0000 7f9586ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:38.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.654+0000 7f9586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:38.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.654+0000 7f9586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:38.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.654+0000 7f9586ffd640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:42:38.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.654+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9598140310 con 0x7f95980a9d70 2024-03-29T11:42:38.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.654+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1865518370 0 0) 0x7f9578004eb0 con 0x7f95981457f0 2024-03-29T11:42:38.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.654+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95981ccf20 con 0x7f95981457f0 2024-03-29T11:42:38.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9590001f40 con 0x7f95981457f0 2024-03-29T11:42:38.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:38.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:38.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 10 monclient(hunting): dump: 2024-03-29T11:42:38.671 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:38.672 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:38.672 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:38.672 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:38.672 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:38.672 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:38.672 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:38.672 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:38.672 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:38.672 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:38.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:38.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 225593926 0 0) 0x7f959813fda0 con 0x7f959813f650 2024-03-29T11:42:38.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9578004eb0 con 0x7f959813f650 2024-03-29T11:42:38.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f958c0018b0 con 0x7f959813f650 2024-03-29T11:42:38.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:38.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:38.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 10 monclient(hunting): dump: 2024-03-29T11:42:38.673 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:38.674 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:38.674 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:38.674 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:38.674 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:38.674 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:38.674 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:38.674 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:38.675 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:38.675 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:38.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 221960616 0 0) 0x7f95981ccf20 con 0x7f95981457f0 2024-03-29T11:42:38.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 >> v1:172.21.15.146:6789/0 conn(0x7f95980a9d70 legacy=0x7f95981407a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:38.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 >> v1:172.21.15.37:6789/0 conn(0x7f959813f650 legacy=0x7f9598140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:38.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:38.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 1 monclient: found mon.c 2024-03-29T11:42:38.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:38.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:38.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959813ff30 con 0x7f95981457f0 2024-03-29T11:42:38.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:38.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:38.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f9586ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:38.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f959c87b640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:42:38.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f959c87b640 10 monclient: _renew_subs 2024-03-29T11:42:38.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f959c87b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:38.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f959c87b640 1 -- 172.21.15.37:0/2068971563 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95981cb340 con 0x7f95981457f0 2024-03-29T11:42:38.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f959c87b640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:38.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f959c87b640 10 monclient: _renew_subs 2024-03-29T11:42:38.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f959c87b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:38.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f959c87b640 1 -- 172.21.15.37:0/2068971563 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9598142c10 con 0x7f95981457f0 2024-03-29T11:42:38.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.655+0000 7f959c87b640 1 librados: init done 2024-03-29T11:42:38.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.656+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9590003c20 con 0x7f95981457f0 2024-03-29T11:42:38.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.656+0000 7f9586ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:38.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.656+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9590004030 con 0x7f95981457f0 2024-03-29T11:42:38.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.656+0000 7f9586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:38.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.656+0000 7f9586ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:38.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.656+0000 7f9586ffd640 10 monclient: dump: 2024-03-29T11:42:38.678 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:38.678 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:38.678 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:38.678 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:38.679 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:38.679 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:38.679 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:38.679 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:38.679 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:38.679 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:38.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.656+0000 7f9594d99640 4 set_mon_vals no callback set 2024-03-29T11:42:38.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.657+0000 7f959c87b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:38.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.657+0000 7f959c87b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:38.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.657+0000 7f959c87b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:38.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.657+0000 7f959c87b640 1 -- 172.21.15.37:0/2068971563 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f957c005080 con 0x7f95981457f0 2024-03-29T11:42:38.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.659+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f9590004190 con 0x7f95981457f0 2024-03-29T11:42:38.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.660+0000 7f959659c640 10 monclient: get_auth_request con 0x7f957806e000 auth_method 0 2024-03-29T11:42:38.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.660+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f9590083cb0 con 0x7f95981457f0 2024-03-29T11:42:38.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.660+0000 7f9586ffd640 10 client.4541.objecter ms_dispatch 0x7f9598000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:38.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.660+0000 7f9586ffd640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:38.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.660+0000 7f9586ffd640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:38.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.660+0000 7f9586ffd640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-29T11:42:38.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.660+0000 7f9586ffd640 10 client.4541.objecter ms_handle_connect 0x7f957806e000 2024-03-29T11:42:38.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.667+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f959004f590 con 0x7f95981457f0 2024-03-29T11:42:38.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.667+0000 7f9586ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:38.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:38.667+0000 7f9586ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:39.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.068+0000 7f959c87b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:39.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.069+0000 7f959c87b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:39.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.069+0000 7f959c87b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:39.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.069+0000 7f959c87b640 1 -- 172.21.15.37:0/2068971563 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f957c005330 con 0x7f95981457f0 2024-03-29T11:42:39.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.070+0000 7f9586ffd640 1 -- 172.21.15.37:0/2068971563 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f95900514c0 con 0x7f95981457f0 2024-03-29T11:42:39.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.070+0000 7f9586ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:39.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.070+0000 7f9586ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:39.074 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:42:39.074 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:39.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.073+0000 7f959c87b640 10 librados: watch_flush enter 2024-03-29T11:42:39.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.073+0000 7f959c87b640 10 librados: watch_flush exit 2024-03-29T11:42:39.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.073+0000 7f959c87b640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-29T11:42:39.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.073+0000 7f959c87b640 10 client.4541.objecter successfully canceled tick 2024-03-29T11:42:39.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.074+0000 7f959c87b640 1 -- 172.21.15.37:0/2068971563 >> v1:172.21.15.146:6816/2899149139 conn(0x7f957806e000 legacy=0x7f95780704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:39.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.074+0000 7f959c87b640 10 monclient: shutdown 2024-03-29T11:42:39.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.074+0000 7f959c87b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:39.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.074+0000 7f959c87b640 1 -- 172.21.15.37:0/2068971563 >> v1:172.21.15.37:6790/0 conn(0x7f95981457f0 legacy=0x7f95981e1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:39.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.074+0000 7f959c87b640 1 -- 172.21.15.37:0/2068971563 shutdown_connections 2024-03-29T11:42:39.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.074+0000 7f959c87b640 1 -- 172.21.15.37:0/2068971563 >> 172.21.15.37:0/2068971563 conn(0x7f9598066a00 msgr2=0x7f9598066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:39.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.074+0000 7f959c87b640 1 -- 172.21.15.37:0/2068971563 shutdown_connections 2024-03-29T11:42:39.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.074+0000 7f959c87b640 1 -- 172.21.15.37:0/2068971563 wait complete. 2024-03-29T11:42:39.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:39.074+0000 7f959c87b640 1 librados: shutdown 2024-03-29T11:42:39.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:45.903 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:46.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.054+0000 7f0840e86640 10 monclient: get_monmap_and_config 2024-03-29T11:42:46.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.054+0000 7f0840e86640 10 monclient: build_initial_monmap 2024-03-29T11:42:46.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.054+0000 7f0840e86640 10 monclient: monmap: 2024-03-29T11:42:46.058 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:46.058 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:46.058 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:42:46.055246+0000 2024-03-29T11:42:46.058 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:42:46.055246+0000 2024-03-29T11:42:46.058 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:46.058 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:46.058 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:46.058 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:42:46.058 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:42:46.058 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:46.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.055+0000 7f0840e86640 1 Processor -- start 2024-03-29T11:42:46.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.055+0000 7f0840e86640 1 -- start start 2024-03-29T11:42:46.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.055+0000 7f0840e86640 10 monclient: init 2024-03-29T11:42:46.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.056+0000 7f0840e86640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:46.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.056+0000 7f0840e86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:46.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.056+0000 7f0840e86640 10 monclient(hunting): picked mon.noname-b con 0x7f083c1457f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:46.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.056+0000 7f0840e86640 10 monclient(hunting): picked mon.noname-c con 0x7f083c13f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:46.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.056+0000 7f0840e86640 10 monclient(hunting): picked mon.noname-a con 0x7f083c0a9d70 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:46.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.056+0000 7f0840e86640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083c0aaef0 con 0x7f083c0a9d70 2024-03-29T11:42:46.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.056+0000 7f0840e86640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083c145bd0 con 0x7f083c13f650 2024-03-29T11:42:46.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.056+0000 7f0840e86640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083c145eb0 con 0x7f083c1457f0 2024-03-29T11:42:46.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.056+0000 7f0840e86640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:46.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.056+0000 7f0840e86640 10 monclient(hunting): authenticate will time out at 1345.505505s 2024-03-29T11:42:46.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.057+0000 7f083a59c640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f083c1457f0 0x7f083c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:36354/0 (socket says 172.21.15.37:36354) 2024-03-29T11:42:46.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.057+0000 7f083a59c640 1 -- 172.21.15.37:0/2971214579 learned_addr learned my addr 172.21.15.37:0/2971214579 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:46.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.058+0000 7f083959a640 1 -- 172.21.15.37:0/2971214579 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129924657 0 0) 0x7f083c145eb0 con 0x7f083c1457f0 2024-03-29T11:42:46.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.058+0000 7f083959a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:46.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.058+0000 7f083959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:46.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.058+0000 7f083959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:46.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.058+0000 7f083959a640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:42:46.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.058+0000 7f083959a640 1 -- 172.21.15.37:0/2971214579 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0824003520 con 0x7f083c1457f0 2024-03-29T11:42:46.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.058+0000 7f083959a640 1 -- 172.21.15.37:0/2971214579 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 249069999 0 0) 0x7f0824003520 con 0x7f083c1457f0 2024-03-29T11:42:46.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.058+0000 7f083959a640 1 -- 172.21.15.37:0/2971214579 >> v1:172.21.15.146:6789/0 conn(0x7f083c13f650 legacy=0x7f083c0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:46.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.058+0000 7f083959a640 1 -- 172.21.15.37:0/2971214579 >> v1:172.21.15.37:6789/0 conn(0x7f083c0a9d70 legacy=0x7f083c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:46.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 10 monclient: _finish_hunting 0 2024-03-29T11:42:46.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 1 monclient: found mon.noname-b 2024-03-29T11:42:46.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:46.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:42:46.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 1 -- 172.21.15.37:0/2971214579 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c146340 con 0x7f083c1457f0 2024-03-29T11:42:46.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 10 monclient: _finish_auth 0 2024-03-29T11:42:46.067 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 10 monclient: _check_auth_tickets 2024-03-29T11:42:46.067 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:46.067 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 1 -- 172.21.15.37:0/2971214579 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0834001a10 con 0x7f083c1457f0 2024-03-29T11:42:46.067 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:46.067 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:46.067 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 10 monclient: dump: 2024-03-29T11:42:46.067 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 1 -- 172.21.15.37:0/2971214579 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0834001be0 con 0x7f083c1457f0 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 1 -- 172.21.15.37:0/2971214579 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0834002030 con 0x7f083c1457f0 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.061+0000 7f083959a640 10 monclient: dump: 2024-03-29T11:42:46.068 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.062+0000 7f0840e86640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.062+0000 7f0840e86640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.062+0000 7f0840e86640 4 set_mon_vals no callback set 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.062+0000 7f0840e86640 10 monclient: shutdown 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.062+0000 7f0840e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:46.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.062+0000 7f0840e86640 1 -- 172.21.15.37:0/2971214579 >> v1:172.21.15.37:6790/0 conn(0x7f083c1457f0 legacy=0x7f083c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.063+0000 7f0840e86640 1 -- 172.21.15.37:0/2971214579 shutdown_connections 2024-03-29T11:42:46.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.063+0000 7f0840e86640 1 -- 172.21.15.37:0/2971214579 >> 172.21.15.37:0/2971214579 conn(0x7f083c066a00 msgr2=0x7f083c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:46.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.063+0000 7f0840e86640 1 -- 172.21.15.37:0/2971214579 shutdown_connections 2024-03-29T11:42:46.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.063+0000 7f0840e86640 1 -- 172.21.15.37:0/2971214579 wait complete. 2024-03-29T11:42:46.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.064+0000 7f0840e86640 10 monclient: build_initial_monmap 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.064+0000 7f0840e86640 10 monclient: monmap: 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.064+0000 7f0840e86640 1 librados: starting msgr at 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.064+0000 7f0840e86640 1 librados: starting objecter 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.064+0000 7f0840e86640 1 Processor -- start 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.065+0000 7f0840e86640 1 -- start start 2024-03-29T11:42:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.065+0000 7f0840e86640 1 librados: setting wanted keys 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.065+0000 7f0840e86640 1 librados: calling monclient init 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.065+0000 7f0840e86640 10 monclient: init 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.065+0000 7f0840e86640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.065+0000 7f0840e86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.065+0000 7f0840e86640 10 monclient(hunting): picked mon.noname-a con 0x7f083c0a9d70 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.065+0000 7f0840e86640 10 monclient(hunting): picked mon.noname-b con 0x7f083c13f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.065+0000 7f0840e86640 10 monclient(hunting): picked mon.noname-c con 0x7f083c1457f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.065+0000 7f0840e86640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083c06f610 con 0x7f083c0a9d70 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.065+0000 7f0840e86640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083c071fa0 con 0x7f083c13f650 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.065+0000 7f0840e86640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083c136e40 con 0x7f083c1457f0 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.065+0000 7f0840e86640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.065+0000 7f0840e86640 10 monclient(hunting): authenticate will time out at 1345.514879s 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.067+0000 7f083ad9d640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f083c1457f0 0x7f083c06e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:37878/0 (socket says 172.21.15.37:37878) 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.067+0000 7f083ad9d640 1 -- 172.21.15.37:0/3741438516 learned_addr learned my addr 172.21.15.37:0/3741438516 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 10 client.?.objecter ms_handle_connect 0x7f083c1457f0 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026311037 0 0) 0x7f083c136e40 con 0x7f083c1457f0 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f081c004eb0 con 0x7f083c1457f0 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1922493882 0 0) 0x7f081c004eb0 con 0x7f083c1457f0 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f083c136e40 con 0x7f083c1457f0 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f082c0018e0 con 0x7f083c1457f0 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 10 monclient(hunting): dump: 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:46.074 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:46.075 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:46.075 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:46.075 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:46.075 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:46.075 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:46.075 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:46.075 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:46.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.068+0000 7f082affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:46.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.069+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2145114914 0 0) 0x7f083c136e40 con 0x7f083c1457f0 2024-03-29T11:42:46.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.069+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 >> v1:172.21.15.146:6789/0 conn(0x7f083c13f650 legacy=0x7f083c06de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.069+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 >> v1:172.21.15.37:6789/0 conn(0x7f083c0a9d70 legacy=0x7f083c137620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:46.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.070+0000 7f082affd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:46.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.070+0000 7f082affd640 1 monclient: found mon.c 2024-03-29T11:42:46.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.070+0000 7f082affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:46.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.070+0000 7f082affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:46.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.070+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c06f0f0 con 0x7f083c1457f0 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.070+0000 7f082affd640 10 monclient: _finish_auth 0 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.070+0000 7f082affd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.070+0000 7f082affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.071+0000 7f0840e86640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.071+0000 7f0840e86640 10 monclient: _renew_subs 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.071+0000 7f0840e86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.071+0000 7f0840e86640 1 -- 172.21.15.37:0/3741438516 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083c06ecb0 con 0x7f083c1457f0 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.071+0000 7f0840e86640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.071+0000 7f0840e86640 10 monclient: _renew_subs 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.071+0000 7f0840e86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.071+0000 7f0840e86640 1 -- 172.21.15.37:0/3741438516 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f083c136720 con 0x7f083c1457f0 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.071+0000 7f0840e86640 1 librados: init done 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.071+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f082c0035f0 con 0x7f083c1457f0 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.071+0000 7f082affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.071+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f082c003a00 con 0x7f083c1457f0 2024-03-29T11:42:46.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.071+0000 7f082bfff640 4 set_mon_vals no callback set 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.072+0000 7f082affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.072+0000 7f082affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.072+0000 7f082affd640 10 monclient: dump: 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.073+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f082c003b60 con 0x7f083c1457f0 2024-03-29T11:42:46.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.074+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f082c084a90 con 0x7f083c1457f0 2024-03-29T11:42:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.074+0000 7f082affd640 10 client.4547.objecter ms_dispatch 0x7f083c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.074+0000 7f082affd640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.074+0000 7f083a59c640 10 monclient: get_auth_request con 0x7f081c06dba0 auth_method 0 2024-03-29T11:42:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.074+0000 7f082affd640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.074+0000 7f082affd640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T11:42:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.075+0000 7f082affd640 10 client.4547.objecter ms_handle_connect 0x7f081c06dba0 2024-03-29T11:42:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.075+0000 7f0840e86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.075+0000 7f0840e86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.075+0000 7f0840e86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.075+0000 7f0840e86640 1 -- 172.21.15.37:0/3741438516 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0820005080 con 0x7f083c1457f0 2024-03-29T11:42:46.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.083+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f082c050370 con 0x7f083c1457f0 2024-03-29T11:42:46.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.083+0000 7f082affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:46.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.083+0000 7f082affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.461+0000 7f0840e86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.461+0000 7f0840e86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.461+0000 7f0840e86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.461+0000 7f0840e86640 1 -- 172.21.15.37:0/3741438516 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0820005330 con 0x7f083c1457f0 2024-03-29T11:42:46.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.463+0000 7f082affd640 1 -- 172.21.15.37:0/3741438516 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f082c0522a0 con 0x7f083c1457f0 2024-03-29T11:42:46.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.463+0000 7f082affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:46.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.463+0000 7f082affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:46.466 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:42:46.466 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:46.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.466+0000 7f0840e86640 10 librados: watch_flush enter 2024-03-29T11:42:46.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.466+0000 7f0840e86640 10 librados: watch_flush exit 2024-03-29T11:42:46.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.466+0000 7f0840e86640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T11:42:46.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.466+0000 7f0840e86640 10 client.4547.objecter successfully canceled tick 2024-03-29T11:42:46.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.466+0000 7f0840e86640 1 -- 172.21.15.37:0/3741438516 >> v1:172.21.15.146:6816/2899149139 conn(0x7f081c06dba0 legacy=0x7f081c070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.466+0000 7f0840e86640 10 monclient: shutdown 2024-03-29T11:42:46.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.466+0000 7f0840e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:46.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.466+0000 7f0840e86640 1 -- 172.21.15.37:0/3741438516 >> v1:172.21.15.37:6790/0 conn(0x7f083c1457f0 legacy=0x7f083c06e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.466+0000 7f0840e86640 1 -- 172.21.15.37:0/3741438516 shutdown_connections 2024-03-29T11:42:46.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.466+0000 7f0840e86640 1 -- 172.21.15.37:0/3741438516 >> 172.21.15.37:0/3741438516 conn(0x7f083c066a00 msgr2=0x7f083c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:46.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.467+0000 7f0840e86640 1 -- 172.21.15.37:0/3741438516 shutdown_connections 2024-03-29T11:42:46.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.467+0000 7f0840e86640 1 -- 172.21.15.37:0/3741438516 wait complete. 2024-03-29T11:42:46.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:46.467+0000 7f0840e86640 1 librados: shutdown 2024-03-29T11:42:46.486 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:53.294 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:53.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.443+0000 7f91c94fb640 10 monclient: get_monmap_and_config 2024-03-29T11:42:53.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.443+0000 7f91c94fb640 10 monclient: build_initial_monmap 2024-03-29T11:42:53.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.443+0000 7f91c94fb640 10 monclient: monmap: 2024-03-29T11:42:53.450 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:53.450 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:53.450 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:42:53.445551+0000 2024-03-29T11:42:53.451 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:42:53.445551+0000 2024-03-29T11:42:53.451 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:53.451 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:53.451 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:53.451 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:42:53.451 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:42:53.451 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:53.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.447+0000 7f91c94fb640 1 Processor -- start 2024-03-29T11:42:53.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.447+0000 7f91c94fb640 1 -- start start 2024-03-29T11:42:53.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.447+0000 7f91c94fb640 10 monclient: init 2024-03-29T11:42:53.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.448+0000 7f91c94fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:53.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.448+0000 7f91c94fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:53.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.448+0000 7f91c94fb640 10 monclient(hunting): picked mon.noname-b con 0x7f91c40a9ac0 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:53.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.448+0000 7f91c94fb640 10 monclient(hunting): picked mon.noname-c con 0x7f91c413ee90 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:53.455 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.450+0000 7f91c94fb640 10 monclient(hunting): picked mon.noname-a con 0x7f91c40af900 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:53.455 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.450+0000 7f91c94fb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c40afd10 con 0x7f91c40af900 2024-03-29T11:42:53.455 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.450+0000 7f91c94fb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c40af6f0 con 0x7f91c413ee90 2024-03-29T11:42:53.455 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.450+0000 7f91c94fb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c40aac80 con 0x7f91c40a9ac0 2024-03-29T11:42:53.455 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.450+0000 7f91c94fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:53.455 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.450+0000 7f91c94fb640 10 monclient(hunting): authenticate will time out at 1352.900828s 2024-03-29T11:42:53.455 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.451+0000 7f91c37fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f91c40af900 0x7f91c4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55099/0 (socket says 172.21.15.37:55099) 2024-03-29T11:42:53.456 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.451+0000 7f91c37fe640 1 -- 172.21.15.37:0/773788708 learned_addr learned my addr 172.21.15.37:0/773788708 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:53.456 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.452+0000 7f91c1ffb640 1 -- 172.21.15.37:0/773788708 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707282708 0 0) 0x7f91c40afd10 con 0x7f91c40af900 2024-03-29T11:42:53.458 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.452+0000 7f91c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:53.458 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.452+0000 7f91c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:53.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.452+0000 7f91c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:53.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.452+0000 7f91c1ffb640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:42:53.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.452+0000 7f91c1ffb640 1 -- 172.21.15.37:0/773788708 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ac003520 con 0x7f91c40af900 2024-03-29T11:42:53.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 1 -- 172.21.15.37:0/773788708 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2037196629 0 0) 0x7f91ac003520 con 0x7f91c40af900 2024-03-29T11:42:53.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 1 -- 172.21.15.37:0/773788708 >> v1:172.21.15.37:6790/0 conn(0x7f91c40a9ac0 legacy=0x7f91c40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 1 -- 172.21.15.37:0/773788708 >> v1:172.21.15.146:6789/0 conn(0x7f91c413ee90 legacy=0x7f91c413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 1 monclient: found mon.noname-a 2024-03-29T11:42:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:42:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 1 -- 172.21.15.37:0/773788708 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c40aaf60 con 0x7f91c40af900 2024-03-29T11:42:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 1 -- 172.21.15.37:0/773788708 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f91bc001a10 con 0x7f91c40af900 2024-03-29T11:42:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:53.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c1ffb640 10 monclient: dump: 2024-03-29T11:42:53.461 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:53.461 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:53.461 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:53.461 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:53.461 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.461 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:53.461 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:53.461 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:53.462 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:53.462 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:53.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c94fb640 5 monclient: authenticate success, global_id 4497 2024-03-29T11:42:53.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.455+0000 7f91c94fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:53.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.461+0000 7f91c1ffb640 1 -- 172.21.15.37:0/773788708 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91bc001c30 con 0x7f91c40af900 2024-03-29T11:42:53.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.461+0000 7f91c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:53.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.461+0000 7f91c1ffb640 1 -- 172.21.15.37:0/773788708 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f91bc001fc0 con 0x7f91c40af900 2024-03-29T11:42:53.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.461+0000 7f91c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.461+0000 7f91c1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:53.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.461+0000 7f91c1ffb640 10 monclient: dump: 2024-03-29T11:42:53.466 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:53.466 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:53.466 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:53.466 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:53.466 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.466 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:53.466 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:53.466 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:53.466 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:53.467 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:53.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.462+0000 7f91c94fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:53.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.462+0000 7f91c94fb640 4 set_mon_vals no callback set 2024-03-29T11:42:53.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.462+0000 7f91c94fb640 10 monclient: shutdown 2024-03-29T11:42:53.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.462+0000 7f91c94fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:53.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.462+0000 7f91c94fb640 1 -- 172.21.15.37:0/773788708 >> v1:172.21.15.37:6789/0 conn(0x7f91c40af900 legacy=0x7f91c4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.463+0000 7f91c94fb640 1 -- 172.21.15.37:0/773788708 shutdown_connections 2024-03-29T11:42:53.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.463+0000 7f91c94fb640 1 -- 172.21.15.37:0/773788708 >> 172.21.15.37:0/773788708 conn(0x7f91c40b35e0 msgr2=0x7f91c40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:53.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.464+0000 7f91c94fb640 1 -- 172.21.15.37:0/773788708 shutdown_connections 2024-03-29T11:42:53.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.464+0000 7f91c94fb640 1 -- 172.21.15.37:0/773788708 wait complete. 2024-03-29T11:42:53.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.465+0000 7f91c94fb640 10 monclient: build_initial_monmap 2024-03-29T11:42:53.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.465+0000 7f91c94fb640 10 monclient: monmap: 2024-03-29T11:42:53.469 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:42:53.469 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:53.469 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:42:53.469 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:42:53.469 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:53.469 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:53.470 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:42:53.470 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:42:53.470 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:42:53.470 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:53.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.465+0000 7f91c94fb640 1 librados: starting msgr at 2024-03-29T11:42:53.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.465+0000 7f91c94fb640 1 librados: starting objecter 2024-03-29T11:42:53.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.466+0000 7f91c94fb640 1 Processor -- start 2024-03-29T11:42:53.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.466+0000 7f91c94fb640 1 -- start start 2024-03-29T11:42:53.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.466+0000 7f91c94fb640 1 librados: setting wanted keys 2024-03-29T11:42:53.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.466+0000 7f91c94fb640 1 librados: calling monclient init 2024-03-29T11:42:53.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.466+0000 7f91c94fb640 10 monclient: init 2024-03-29T11:42:53.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.467+0000 7f91c94fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:53.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.467+0000 7f91c94fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:53.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.470+0000 7f91c94fb640 10 monclient(hunting): picked mon.noname-b con 0x7f91c40a9ac0 addr v1:172.21.15.146:6789/0 2024-03-29T11:42:53.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.473+0000 7f91c2ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f91c40a9ac0 0x7f91c4071440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:41445/0 (socket says 172.21.15.37:41445) 2024-03-29T11:42:53.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.473+0000 7f91c94fb640 10 monclient(hunting): picked mon.noname-a con 0x7f91c40af900 addr v1:172.21.15.37:6789/0 2024-03-29T11:42:53.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.477+0000 7f91c94fb640 10 monclient(hunting): picked mon.noname-c con 0x7f91c413ee90 addr v1:172.21.15.37:6790/0 2024-03-29T11:42:53.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.477+0000 7f91c94fb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c40704b0 con 0x7f91c40af900 2024-03-29T11:42:53.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.477+0000 7f91c94fb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c4070720 con 0x7f91c40a9ac0 2024-03-29T11:42:53.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.477+0000 7f91c94fb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c4070a00 con 0x7f91c413ee90 2024-03-29T11:42:53.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.477+0000 7f91c94fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:53.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.477+0000 7f91c94fb640 10 monclient(hunting): authenticate will time out at 1352.927012s 2024-03-29T11:42:53.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.477+0000 7f91c27fc640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f91c40af900 0x7f91c4071b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55485/0 (socket says 172.21.15.37:55485) 2024-03-29T11:42:53.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.477+0000 7f91c27fc640 1 -- 172.21.15.37:0/3517697398 learned_addr learned my addr 172.21.15.37:0/3517697398 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:42:53.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 10 client.?.objecter ms_handle_connect 0x7f91c40af900 2024-03-29T11:42:53.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:53.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 10 client.?.objecter ms_handle_connect 0x7f91c413ee90 2024-03-29T11:42:53.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:53.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 10 client.?.objecter ms_handle_connect 0x7f91c40a9ac0 2024-03-29T11:42:53.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:53.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813301687 0 0) 0x7f91c4070a00 con 0x7f91c413ee90 2024-03-29T11:42:53.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:53.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:53.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:53.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:42:53.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b00092c0 con 0x7f91c413ee90 2024-03-29T11:42:53.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229186973 0 0) 0x7f91c40704b0 con 0x7f91c40af900 2024-03-29T11:42:53.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:53.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:53.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:53.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:42:53.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.479+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c4070a00 con 0x7f91c40af900 2024-03-29T11:42:53.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.480+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226737959 0 0) 0x7f91c4070720 con 0x7f91c40a9ac0 2024-03-29T11:42:53.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.480+0000 7f91ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:53.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.480+0000 7f91ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:53.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.480+0000 7f91ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:53.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.480+0000 7f91ab7fe640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:42:53.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.480+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c40704b0 con 0x7f91c40a9ac0 2024-03-29T11:42:53.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.480+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 797321825 0 0) 0x7f91b00092c0 con 0x7f91c413ee90 2024-03-29T11:42:53.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91b000ab30 con 0x7f91c413ee90 2024-03-29T11:42:53.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2360187209 0 0) 0x7f91c4070a00 con 0x7f91c40af900 2024-03-29T11:42:53.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91b00092c0 con 0x7f91c40af900 2024-03-29T11:42:53.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f91bc001da0 con 0x7f91c413ee90 2024-03-29T11:42:53.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:53.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient(hunting): dump: 2024-03-29T11:42:53.492 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:53.492 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:53.492 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:53.493 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:53.493 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.493 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:53.493 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:53.493 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:53.493 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:53.493 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:53.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:53.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f91b4001740 con 0x7f91c40af900 2024-03-29T11:42:53.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:53.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient(hunting): dump: 2024-03-29T11:42:53.494 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:53.494 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:53.494 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:53.494 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:53.495 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.495 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:53.495 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:53.495 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:53.495 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:53.495 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:53.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2626663283 0 0) 0x7f91b000ab30 con 0x7f91c413ee90 2024-03-29T11:42:53.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 >> v1:172.21.15.146:6789/0 conn(0x7f91c40a9ac0 legacy=0x7f91c4071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 >> v1:172.21.15.37:6789/0 conn(0x7f91c40af900 legacy=0x7f91c4071b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:53.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 1 monclient: found mon.c 2024-03-29T11:42:53.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:53.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:53.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c4070ce0 con 0x7f91c413ee90 2024-03-29T11:42:53.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:53.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:53.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:53.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91c94fb640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:42:53.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91c94fb640 10 monclient: _renew_subs 2024-03-29T11:42:53.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91c94fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:53.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91c94fb640 1 -- 172.21.15.37:0/3517697398 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c412d3b0 con 0x7f91c413ee90 2024-03-29T11:42:53.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91c94fb640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:53.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91c94fb640 10 monclient: _renew_subs 2024-03-29T11:42:53.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91c94fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:53.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91c94fb640 1 -- 172.21.15.37:0/3517697398 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c406ebf0 con 0x7f91c413ee90 2024-03-29T11:42:53.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91c94fb640 1 librados: init done 2024-03-29T11:42:53.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91bc004cc0 con 0x7f91c413ee90 2024-03-29T11:42:53.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:53.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f91bc004fd0 con 0x7f91c413ee90 2024-03-29T11:42:53.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:53.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91ab7fe640 10 monclient: dump: 2024-03-29T11:42:53.497 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:42:53.497 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:42:53.497 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:53.498 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:42:53.498 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.498 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:42:53.498 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:42:53.498 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:42:53.498 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:42:53.498 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:42:53.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.481+0000 7f91c17fa640 4 set_mon_vals no callback set 2024-03-29T11:42:53.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.483+0000 7f91c94fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:53.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.483+0000 7f91c94fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:53.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.483+0000 7f91c94fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:53.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.483+0000 7f91c94fb640 1 -- 172.21.15.37:0/3517697398 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a4005080 con 0x7f91c413ee90 2024-03-29T11:42:53.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.484+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f91bc01afa0 con 0x7f91c413ee90 2024-03-29T11:42:53.499 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.484+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f91bc084dd0 con 0x7f91c413ee90 2024-03-29T11:42:53.499 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.484+0000 7f91ab7fe640 10 client.4550.objecter ms_dispatch 0x7f91c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:53.499 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.484+0000 7f91ab7fe640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:53.499 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.484+0000 7f91ab7fe640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:53.499 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.485+0000 7f91ab7fe640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-29T11:42:53.499 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.485+0000 7f91c2ffd640 10 monclient: get_auth_request con 0x7f91b0076810 auth_method 0 2024-03-29T11:42:53.499 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.486+0000 7f91ab7fe640 10 client.4550.objecter ms_handle_connect 0x7f91b0076810 2024-03-29T11:42:53.502 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.498+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f91bc0506b0 con 0x7f91c413ee90 2024-03-29T11:42:53.502 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.498+0000 7f91ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:53.503 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.498+0000 7f91ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:53.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.831+0000 7f91c94fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:53.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.831+0000 7f91c94fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:53.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.831+0000 7f91c94fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:42:53.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.831+0000 7f91c94fb640 1 -- 172.21.15.37:0/3517697398 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91a4005330 con 0x7f91c413ee90 2024-03-29T11:42:53.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.832+0000 7f91ab7fe640 1 -- 172.21.15.37:0/3517697398 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f91bc0525e0 con 0x7f91c413ee90 2024-03-29T11:42:53.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.832+0000 7f91ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:53.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.832+0000 7f91ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:53.837 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:42:53.837 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:53.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.835+0000 7f91c94fb640 10 librados: watch_flush enter 2024-03-29T11:42:53.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.835+0000 7f91c94fb640 10 librados: watch_flush exit 2024-03-29T11:42:53.840 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.835+0000 7f91c94fb640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-29T11:42:53.840 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.835+0000 7f91c94fb640 10 client.4550.objecter successfully canceled tick 2024-03-29T11:42:53.840 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.835+0000 7f91c94fb640 1 -- 172.21.15.37:0/3517697398 >> v1:172.21.15.146:6816/2899149139 conn(0x7f91b0076810 legacy=0x7f91b0078cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.840 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.835+0000 7f91c94fb640 10 monclient: shutdown 2024-03-29T11:42:53.840 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.835+0000 7f91c94fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:53.840 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.835+0000 7f91c94fb640 1 -- 172.21.15.37:0/3517697398 >> v1:172.21.15.37:6790/0 conn(0x7f91c413ee90 legacy=0x7f91c41dfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.841 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.837+0000 7f91c94fb640 1 -- 172.21.15.37:0/3517697398 shutdown_connections 2024-03-29T11:42:53.841 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.837+0000 7f91c94fb640 1 -- 172.21.15.37:0/3517697398 >> 172.21.15.37:0/3517697398 conn(0x7f91c40b35e0 msgr2=0x7f91c413b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:53.841 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.837+0000 7f91c94fb640 1 -- 172.21.15.37:0/3517697398 shutdown_connections 2024-03-29T11:42:53.841 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.837+0000 7f91c94fb640 1 -- 172.21.15.37:0/3517697398 wait complete. 2024-03-29T11:42:53.841 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:42:53.837+0000 7f91c94fb640 1 librados: shutdown 2024-03-29T11:42:53.857 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:00.664 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:00.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.808+0000 7f6ce5ad4640 10 monclient: get_monmap_and_config 2024-03-29T11:43:00.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.808+0000 7f6ce5ad4640 10 monclient: build_initial_monmap 2024-03-29T11:43:00.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.808+0000 7f6ce5ad4640 10 monclient: monmap: 2024-03-29T11:43:00.813 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:00.813 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:00.813 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:43:00.810611+0000 2024-03-29T11:43:00.813 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:43:00.810611+0000 2024-03-29T11:43:00.813 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:00.813 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:00.813 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:00.813 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:43:00.813 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:43:00.813 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:00.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.809+0000 7f6ce5ad4640 1 Processor -- start 2024-03-29T11:43:00.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.810+0000 7f6ce5ad4640 1 -- start start 2024-03-29T11:43:00.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.810+0000 7f6ce5ad4640 10 monclient: init 2024-03-29T11:43:00.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.810+0000 7f6ce5ad4640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:00.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.810+0000 7f6ce5ad4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:00.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.810+0000 7f6ce5ad4640 10 monclient(hunting): picked mon.noname-c con 0x7f6cd8006e30 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:00.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.810+0000 7f6ce5ad4640 10 monclient(hunting): picked mon.noname-b con 0x7f6cd8035e00 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:00.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.811+0000 7f6ce5ad4640 10 monclient(hunting): picked mon.noname-a con 0x7f6cd80c9670 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:00.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.811+0000 7f6ce5ad4640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd8035970 con 0x7f6cd80c9670 2024-03-29T11:43:00.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.811+0000 7f6ce5ad4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd8036280 con 0x7f6cd8006e30 2024-03-29T11:43:00.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.811+0000 7f6ce5ad4640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd8035640 con 0x7f6cd8035e00 2024-03-29T11:43:00.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.811+0000 7f6ce5ad4640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:00.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.811+0000 7f6ce5ad4640 10 monclient(hunting): authenticate will time out at 1360.260991s 2024-03-29T11:43:00.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.811+0000 7f6cdeffd640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f6cd8035e00 0x7f6cd80c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:43132/0 (socket says 172.21.15.37:43132) 2024-03-29T11:43:00.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.811+0000 7f6cdeffd640 1 -- 172.21.15.37:0/3311857324 learned_addr learned my addr 172.21.15.37:0/3311857324 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:00.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.815+0000 7f6cde7fc640 1 -- 172.21.15.37:0/3311857324 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236429130 0 0) 0x7f6cd8035640 con 0x7f6cd8035e00 2024-03-29T11:43:00.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.815+0000 7f6cde7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:00.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.815+0000 7f6cde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:00.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.815+0000 7f6cde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:00.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.815+0000 7f6cde7fc640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:43:00.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.815+0000 7f6cde7fc640 1 -- 172.21.15.37:0/3311857324 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cd0003520 con 0x7f6cd8035e00 2024-03-29T11:43:00.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.815+0000 7f6cde7fc640 1 -- 172.21.15.37:0/3311857324 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254423024 0 0) 0x7f6cd8036280 con 0x7f6cd8006e30 2024-03-29T11:43:00.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.815+0000 7f6cde7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:00.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.815+0000 7f6cde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:00.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.815+0000 7f6cde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:00.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.815+0000 7f6cde7fc640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:43:00.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.815+0000 7f6cde7fc640 1 -- 172.21.15.37:0/3311857324 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cd8035640 con 0x7f6cd8006e30 2024-03-29T11:43:00.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 1 -- 172.21.15.37:0/3311857324 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3881410126 0 0) 0x7f6cd0003520 con 0x7f6cd8035e00 2024-03-29T11:43:00.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 1 -- 172.21.15.37:0/3311857324 >> v1:172.21.15.146:6789/0 conn(0x7f6cd8006e30 legacy=0x7f6cd80cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:00.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 1 -- 172.21.15.37:0/3311857324 >> v1:172.21.15.37:6789/0 conn(0x7f6cd80c9670 legacy=0x7f6cd80d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:00.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:00.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 1 monclient: found mon.noname-b 2024-03-29T11:43:00.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:00.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:43:00.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 1 -- 172.21.15.37:0/3311857324 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd8034e20 con 0x7f6cd8035e00 2024-03-29T11:43:00.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:00.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:00.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:00.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6ce5ad4640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:43:00.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6ce5ad4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:00.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 1 -- 172.21.15.37:0/3311857324 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6ce004f730 con 0x7f6cd8035e00 2024-03-29T11:43:00.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:00.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:00.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 10 monclient: dump: 2024-03-29T11:43:00.822 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:00.822 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:00.822 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:00.822 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:00.822 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:00.822 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6ce5ad4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 1 -- 172.21.15.37:0/3311857324 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ce0075f10 con 0x7f6cd8035e00 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6ce5ad4640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6ce5ad4640 4 set_mon_vals no callback set 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 1 -- 172.21.15.37:0/3311857324 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6ce00758b0 con 0x7f6cd8035e00 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.816+0000 7f6cde7fc640 10 monclient: dump: 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:00.823 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:00.824 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:00.824 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:00.824 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:00.824 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:00.824 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:00.824 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:00.824 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:00.824 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:00.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.817+0000 7f6ce5ad4640 10 monclient: shutdown 2024-03-29T11:43:00.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.817+0000 7f6ce5ad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:00.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.817+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/3311857324 >> v1:172.21.15.37:6790/0 conn(0x7f6cd8035e00 legacy=0x7f6cd80c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:00.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.817+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/3311857324 shutdown_connections 2024-03-29T11:43:00.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.817+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/3311857324 >> 172.21.15.37:0/3311857324 conn(0x7f6cd8005c00 msgr2=0x7f6cd8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:00.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.817+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/3311857324 shutdown_connections 2024-03-29T11:43:00.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.818+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/3311857324 wait complete. 2024-03-29T11:43:00.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.818+0000 7f6ce5ad4640 10 monclient: build_initial_monmap 2024-03-29T11:43:00.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.818+0000 7f6ce5ad4640 10 monclient: monmap: 2024-03-29T11:43:00.828 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:00.828 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:00.828 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:43:00.828 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.819+0000 7f6ce5ad4640 1 librados: starting msgr at 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.819+0000 7f6ce5ad4640 1 librados: starting objecter 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.819+0000 7f6ce5ad4640 1 Processor -- start 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.819+0000 7f6ce5ad4640 1 -- start start 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.819+0000 7f6ce5ad4640 1 librados: setting wanted keys 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.819+0000 7f6ce5ad4640 1 librados: calling monclient init 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.819+0000 7f6ce5ad4640 10 monclient: init 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.820+0000 7f6ce5ad4640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:00.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.820+0000 7f6ce5ad4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:00.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.820+0000 7f6ce5ad4640 10 monclient(hunting): picked mon.noname-c con 0x7f6cd8006e30 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:00.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.820+0000 7f6ce5ad4640 10 monclient(hunting): picked mon.noname-a con 0x7f6cd8035e00 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:00.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.820+0000 7f6ce5ad4640 10 monclient(hunting): picked mon.noname-b con 0x7f6cd80c9670 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:00.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.820+0000 7f6ce5ad4640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd81872b0 con 0x7f6cd8035e00 2024-03-29T11:43:00.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.820+0000 7f6ce5ad4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd802f2b0 con 0x7f6cd80c9670 2024-03-29T11:43:00.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.820+0000 7f6ce5ad4640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd81832a0 con 0x7f6cd8006e30 2024-03-29T11:43:00.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.820+0000 7f6ce5ad4640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:00.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.820+0000 7f6ce5ad4640 10 monclient(hunting): authenticate will time out at 1360.270770s 2024-03-29T11:43:00.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.821+0000 7f6cdf7fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f6cd8006e30 0x7f6cd8186320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:54596/0 (socket says 172.21.15.37:54596) 2024-03-29T11:43:00.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.821+0000 7f6cdf7fe640 1 -- 172.21.15.37:0/2213885383 learned_addr learned my addr 172.21.15.37:0/2213885383 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:00.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.821+0000 7f6ccbfff640 10 client.?.objecter ms_handle_connect 0x7f6cd8006e30 2024-03-29T11:43:00.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.821+0000 7f6ccbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:00.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.823+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722640434 0 0) 0x7f6cd81832a0 con 0x7f6cd8006e30 2024-03-29T11:43:00.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.823+0000 7f6ccbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:00.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.823+0000 7f6ccbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:00.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.823+0000 7f6ccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:00.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.823+0000 7f6ccbfff640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:43:00.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.823+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc4004eb0 con 0x7f6cd8006e30 2024-03-29T11:43:00.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.824+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1191896740 0 0) 0x7f6cc4004eb0 con 0x7f6cd8006e30 2024-03-29T11:43:00.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.824+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cd81832a0 con 0x7f6cd8006e30 2024-03-29T11:43:00.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.824+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6cd4001a10 con 0x7f6cd8006e30 2024-03-29T11:43:00.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.824+0000 7f6ccbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:00.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.824+0000 7f6ccbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:00.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.824+0000 7f6ccbfff640 10 monclient(hunting): dump: 2024-03-29T11:43:00.832 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:00.832 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:00.832 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:00.832 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:00.832 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:00.832 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:00.832 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:00.832 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:00.832 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:00.833 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:00.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.824+0000 7f6ccbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:00.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.825+0000 7f6ccbfff640 10 client.?.objecter ms_handle_connect 0x7f6cd8035e00 2024-03-29T11:43:00.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.825+0000 7f6ccbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:00.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.825+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2975177394 0 0) 0x7f6cd81832a0 con 0x7f6cd8006e30 2024-03-29T11:43:00.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.825+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 >> v1:172.21.15.146:6789/0 conn(0x7f6cd80c9670 legacy=0x7f6cd81800a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:00.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.827+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 >> v1:172.21.15.37:6789/0 conn(0x7f6cd8035e00 legacy=0x7f6cd802fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:00.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.827+0000 7f6ccbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:00.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.827+0000 7f6ccbfff640 1 monclient: found mon.c 2024-03-29T11:43:00.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.827+0000 7f6ccbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:00.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.827+0000 7f6ccbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:00.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.827+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd8030150 con 0x7f6cd8006e30 2024-03-29T11:43:00.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.827+0000 7f6ccbfff640 10 monclient: _finish_auth 0 2024-03-29T11:43:00.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.827+0000 7f6ccbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:00.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.827+0000 7f6ccbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:00.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ce5ad4640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:43:00.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ce5ad4640 10 monclient: _renew_subs 2024-03-29T11:43:00.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ce5ad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:00.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/2213885383 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd81807b0 con 0x7f6cd8006e30 2024-03-29T11:43:00.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ce5ad4640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:00.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ce5ad4640 10 monclient: _renew_subs 2024-03-29T11:43:00.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ce5ad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:00.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/2213885383 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd8033d20 con 0x7f6cd8006e30 2024-03-29T11:43:00.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ce5ad4640 1 librados: init done 2024-03-29T11:43:00.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cd4001d10 con 0x7f6cd8006e30 2024-03-29T11:43:00.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ccbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:00.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6cd4003a60 con 0x7f6cd8006e30 2024-03-29T11:43:00.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ccbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:00.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ccbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:00.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.828+0000 7f6ccbfff640 10 monclient: dump: 2024-03-29T11:43:00.836 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:00.836 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:00.836 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:00.836 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:00.836 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:00.836 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:00.836 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:00.836 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:00.836 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:00.836 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:00.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.829+0000 7f6cddffb640 4 set_mon_vals no callback set 2024-03-29T11:43:00.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.830+0000 7f6ce5ad4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:00.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.830+0000 7f6ce5ad4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:00.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.830+0000 7f6ce5ad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:00.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.830+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/2213885383 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc0005080 con 0x7f6cd8006e30 2024-03-29T11:43:00.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.831+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f6cd4003bc0 con 0x7f6cd8006e30 2024-03-29T11:43:00.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.832+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f6cd4084b50 con 0x7f6cd8006e30 2024-03-29T11:43:00.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.832+0000 7f6ccbfff640 10 client.4556.objecter ms_dispatch 0x7f6ce0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:00.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.832+0000 7f6ccbfff640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:00.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.832+0000 7f6ccbfff640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:00.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.832+0000 7f6ccbfff640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T11:43:00.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.832+0000 7f6cdeffd640 10 monclient: get_auth_request con 0x7f6cc406dae0 auth_method 0 2024-03-29T11:43:00.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.833+0000 7f6ccbfff640 10 client.4556.objecter ms_handle_connect 0x7f6cc406dae0 2024-03-29T11:43:00.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.838+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6cd4050430 con 0x7f6cd8006e30 2024-03-29T11:43:00.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.839+0000 7f6ccbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:00.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:00.839+0000 7f6ccbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:01.183 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.176+0000 7f6ce5ad4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:01.183 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.176+0000 7f6ce5ad4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:01.183 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.176+0000 7f6ce5ad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:01.183 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.176+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/2213885383 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cc0005330 con 0x7f6cd8006e30 2024-03-29T11:43:01.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.180+0000 7f6ccbfff640 1 -- 172.21.15.37:0/2213885383 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f6cd4052360 con 0x7f6cd8006e30 2024-03-29T11:43:01.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.180+0000 7f6ccbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:01.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.180+0000 7f6ccbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:01.188 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:43:01.188 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:01.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.187+0000 7f6ce5ad4640 10 librados: watch_flush enter 2024-03-29T11:43:01.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.188+0000 7f6ce5ad4640 10 librados: watch_flush exit 2024-03-29T11:43:01.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.188+0000 7f6ce5ad4640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T11:43:01.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.188+0000 7f6ce5ad4640 10 client.4556.objecter successfully canceled tick 2024-03-29T11:43:01.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.188+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/2213885383 >> v1:172.21.15.146:6816/2899149139 conn(0x7f6cc406dae0 legacy=0x7f6cc406ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:01.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.188+0000 7f6ce5ad4640 10 monclient: shutdown 2024-03-29T11:43:01.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.188+0000 7f6ce5ad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:01.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.188+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/2213885383 >> v1:172.21.15.37:6790/0 conn(0x7f6cd8006e30 legacy=0x7f6cd8186320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:01.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.188+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/2213885383 shutdown_connections 2024-03-29T11:43:01.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.188+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/2213885383 >> 172.21.15.37:0/2213885383 conn(0x7f6cd8005c00 msgr2=0x7f6cd80c7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.190+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/2213885383 shutdown_connections 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.190+0000 7f6ce5ad4640 1 -- 172.21.15.37:0/2213885383 wait complete. 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:01.190+0000 7f6ce5ad4640 1 librados: shutdown 2024-03-29T11:43:01.212 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:08.019 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:08.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.170+0000 7f8b8641e640 10 monclient: get_monmap_and_config 2024-03-29T11:43:08.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.170+0000 7f8b8641e640 10 monclient: build_initial_monmap 2024-03-29T11:43:08.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.170+0000 7f8b8641e640 10 monclient: monmap: 2024-03-29T11:43:08.174 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:08.174 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:08.174 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:43:08.171088+0000 2024-03-29T11:43:08.174 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:43:08.171088+0000 2024-03-29T11:43:08.174 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:08.174 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:08.175 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:08.175 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:43:08.175 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:43:08.175 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:08.175 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.171+0000 7f8b8641e640 1 Processor -- start 2024-03-29T11:43:08.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.171+0000 7f8b8641e640 1 -- start start 2024-03-29T11:43:08.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.171+0000 7f8b8641e640 10 monclient: init 2024-03-29T11:43:08.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.172+0000 7f8b8641e640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:08.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.172+0000 7f8b8641e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:08.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.172+0000 7f8b8641e640 10 monclient(hunting): picked mon.noname-b con 0x7f8b801457f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:08.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.172+0000 7f8b7ffff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f8b801457f0 0x7f8b8013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:42916/0 (socket says 172.21.15.37:42916) 2024-03-29T11:43:08.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.172+0000 7f8b7ffff640 1 -- 172.21.15.37:0/4021914818 learned_addr learned my addr 172.21.15.37:0/4021914818 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:08.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.172+0000 7f8b8641e640 10 monclient(hunting): picked mon.noname-c con 0x7f8b8013f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:08.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.174+0000 7f8b8641e640 10 monclient(hunting): picked mon.noname-a con 0x7f8b800a9d70 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:08.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.174+0000 7f8b8641e640 1 -- 172.21.15.37:0/4021914818 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b800aaef0 con 0x7f8b800a9d70 2024-03-29T11:43:08.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.174+0000 7f8b8641e640 1 -- 172.21.15.37:0/4021914818 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b80145bd0 con 0x7f8b8013f650 2024-03-29T11:43:08.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.174+0000 7f8b8641e640 1 -- 172.21.15.37:0/4021914818 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b80145eb0 con 0x7f8b801457f0 2024-03-29T11:43:08.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.174+0000 7f8b8641e640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:08.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.174+0000 7f8b8641e640 10 monclient(hunting): authenticate will time out at 1367.623386s 2024-03-29T11:43:08.183 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.178+0000 7f8b7effd640 1 -- 172.21.15.37:0/4021914818 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978442247 0 0) 0x7f8b80145eb0 con 0x7f8b801457f0 2024-03-29T11:43:08.183 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.178+0000 7f8b7effd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:08.183 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 1 -- 172.21.15.37:0/4021914818 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b68003520 con 0x7f8b801457f0 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 1 -- 172.21.15.37:0/4021914818 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812552306 0 0) 0x7f8b80145bd0 con 0x7f8b8013f650 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 1 -- 172.21.15.37:0/4021914818 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b80145eb0 con 0x7f8b8013f650 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 1 -- 172.21.15.37:0/4021914818 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1432907663 0 0) 0x7f8b68003520 con 0x7f8b801457f0 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 1 -- 172.21.15.37:0/4021914818 >> v1:172.21.15.146:6789/0 conn(0x7f8b8013f650 legacy=0x7f8b800b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 1 -- 172.21.15.37:0/4021914818 >> v1:172.21.15.37:6789/0 conn(0x7f8b800a9d70 legacy=0x7f8b8014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 1 monclient: found mon.noname-b 2024-03-29T11:43:08.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 1 -- 172.21.15.37:0/4021914818 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b80146340 con 0x7f8b801457f0 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient: _finish_auth 0 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 1 -- 172.21.15.37:0/4021914818 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8b70001690 con 0x7f8b801457f0 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b8641e640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b8641e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient: dump: 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:08.185 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b8641e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 1 -- 172.21.15.37:0/4021914818 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b700018b0 con 0x7f8b801457f0 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b7effd640 1 -- 172.21.15.37:0/4021914818 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8b70001c40 con 0x7f8b801457f0 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b8641e640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.179+0000 7f8b8641e640 4 set_mon_vals no callback set 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.180+0000 7f8b7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.180+0000 7f8b7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.180+0000 7f8b7effd640 10 monclient: dump: 2024-03-29T11:43:08.186 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:08.187 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:08.187 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:08.187 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:08.187 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:08.187 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:08.187 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:08.187 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:08.187 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:08.187 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:08.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.180+0000 7f8b8641e640 10 monclient: shutdown 2024-03-29T11:43:08.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.180+0000 7f8b8641e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:08.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.180+0000 7f8b8641e640 1 -- 172.21.15.37:0/4021914818 >> v1:172.21.15.37:6790/0 conn(0x7f8b801457f0 legacy=0x7f8b8013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.181+0000 7f8b8641e640 1 -- 172.21.15.37:0/4021914818 shutdown_connections 2024-03-29T11:43:08.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.181+0000 7f8b8641e640 1 -- 172.21.15.37:0/4021914818 >> 172.21.15.37:0/4021914818 conn(0x7f8b80066a00 msgr2=0x7f8b80066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:08.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.181+0000 7f8b8641e640 1 -- 172.21.15.37:0/4021914818 shutdown_connections 2024-03-29T11:43:08.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.181+0000 7f8b8641e640 1 -- 172.21.15.37:0/4021914818 wait complete. 2024-03-29T11:43:08.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.182+0000 7f8b8641e640 10 monclient: build_initial_monmap 2024-03-29T11:43:08.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.182+0000 7f8b8641e640 10 monclient: monmap: 2024-03-29T11:43:08.190 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:08.190 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:08.190 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:43:08.190 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:43:08.190 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:08.190 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:08.190 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:08.190 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:43:08.190 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:43:08.190 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:08.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.183+0000 7f8b8641e640 1 librados: starting msgr at 2024-03-29T11:43:08.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.183+0000 7f8b8641e640 1 librados: starting objecter 2024-03-29T11:43:08.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.183+0000 7f8b8641e640 1 Processor -- start 2024-03-29T11:43:08.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.183+0000 7f8b8641e640 1 -- start start 2024-03-29T11:43:08.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.183+0000 7f8b8641e640 1 librados: setting wanted keys 2024-03-29T11:43:08.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.183+0000 7f8b8641e640 1 librados: calling monclient init 2024-03-29T11:43:08.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.184+0000 7f8b8641e640 10 monclient: init 2024-03-29T11:43:08.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.184+0000 7f8b8641e640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:08.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.185+0000 7f8b8641e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:08.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.188+0000 7f8b8641e640 10 monclient(hunting): picked mon.noname-b con 0x7f8b800a9d70 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:08.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.188+0000 7f8b7ffff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f8b800a9d70 0x7f8b80138a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:54267/0 (socket says 172.21.15.37:54267) 2024-03-29T11:43:08.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.188+0000 7f8b7ffff640 1 -- 172.21.15.37:0/3498363524 learned_addr learned my addr 172.21.15.37:0/3498363524 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:08.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.188+0000 7f8b8641e640 10 monclient(hunting): picked mon.noname-c con 0x7f8b8013f650 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:08.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.189+0000 7f8b7cff9640 10 client.?.objecter ms_handle_connect 0x7f8b800a9d70 2024-03-29T11:43:08.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.189+0000 7f8b7cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:08.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.189+0000 7f8b8641e640 10 monclient(hunting): picked mon.noname-a con 0x7f8b801457f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:08.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.189+0000 7f8b8641e640 1 -- 172.21.15.37:0/3498363524 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b801391a0 con 0x7f8b801457f0 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.189+0000 7f8b8641e640 1 -- 172.21.15.37:0/3498363524 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8013b0b0 con 0x7f8b800a9d70 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.189+0000 7f8b8641e640 1 -- 172.21.15.37:0/3498363524 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8013ba60 con 0x7f8b8013f650 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.189+0000 7f8b8641e640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.189+0000 7f8b8641e640 10 monclient(hunting): authenticate will time out at 1367.638871s 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.190+0000 7f8b7cff9640 10 client.?.objecter ms_handle_connect 0x7f8b8013f650 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.190+0000 7f8b7cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.190+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858923228 0 0) 0x7f8b8013b0b0 con 0x7f8b800a9d70 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.190+0000 7f8b7cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.190+0000 7f8b7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.190+0000 7f8b7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.190+0000 7f8b7cff9640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.190+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b6c004eb0 con 0x7f8b800a9d70 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.190+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471720580 0 0) 0x7f8b8013ba60 con 0x7f8b8013f650 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.190+0000 7f8b7cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:08.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.190+0000 7f8b7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.190+0000 7f8b7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.190+0000 7f8b7cff9640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.191+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b8013b0b0 con 0x7f8b8013f650 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.191+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3798100000 0 0) 0x7f8b6c004eb0 con 0x7f8b800a9d70 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.191+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b8013ba60 con 0x7f8b800a9d70 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.191+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8b70001c80 con 0x7f8b800a9d70 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.191+0000 7f8b7cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.191+0000 7f8b7cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.191+0000 7f8b7cff9640 10 monclient(hunting): dump: 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:08.199 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:08.200 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:08.200 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:08.200 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:08.200 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:08.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.191+0000 7f8b7cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:08.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.191+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1741230217 0 0) 0x7f8b8013ba60 con 0x7f8b800a9d70 2024-03-29T11:43:08.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.191+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 >> v1:172.21.15.37:6790/0 conn(0x7f8b8013f650 legacy=0x7f8b801d7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.191+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 >> v1:172.21.15.37:6789/0 conn(0x7f8b801457f0 legacy=0x7f8b80134f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:08.204 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.192+0000 7f8b7cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:43:08.204 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.192+0000 7f8b7cff9640 1 monclient: found mon.b 2024-03-29T11:43:08.204 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.192+0000 7f8b7cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:08.204 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.193+0000 7f8b7cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:08.204 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.193+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8013b4d0 con 0x7f8b800a9d70 2024-03-29T11:43:08.204 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.193+0000 7f8b7cff9640 10 monclient: _finish_auth 0 2024-03-29T11:43:08.204 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.193+0000 7f8b7cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:43:08.204 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.193+0000 7f8b7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:08.204 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.193+0000 7f8b8641e640 5 monclient: authenticate success, global_id 4468 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.193+0000 7f8b8641e640 10 monclient: _renew_subs 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.193+0000 7f8b8641e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.193+0000 7f8b8641e640 1 -- 172.21.15.37:0/3498363524 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b80139730 con 0x7f8b800a9d70 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.193+0000 7f8b8641e640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.193+0000 7f8b8641e640 10 monclient: _renew_subs 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.193+0000 7f8b8641e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.193+0000 7f8b8641e640 1 -- 172.21.15.37:0/3498363524 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b801d7e90 con 0x7f8b800a9d70 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.193+0000 7f8b8641e640 1 librados: init done 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.195+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b70004b00 con 0x7f8b800a9d70 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.195+0000 7f8b7cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.195+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8b70004ed0 con 0x7f8b800a9d70 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.195+0000 7f8b7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.195+0000 7f8b7cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.195+0000 7f8b7cff9640 10 monclient: dump: 2024-03-29T11:43:08.205 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.195+0000 7f8b7dffb640 4 set_mon_vals no callback set 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.197+0000 7f8b667fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.197+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f8b700050f0 con 0x7f8b800a9d70 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.198+0000 7f8b667fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.198+0000 7f8b667fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:08.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.198+0000 7f8b667fc640 1 -- 172.21.15.37:0/3498363524 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b60005080 con 0x7f8b800a9d70 2024-03-29T11:43:08.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.198+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f8b70084ba0 con 0x7f8b800a9d70 2024-03-29T11:43:08.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.198+0000 7f8b7cff9640 10 client.4468.objecter ms_dispatch 0x7f8b80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:08.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.198+0000 7f8b7cff9640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:08.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.198+0000 7f8b7cff9640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:08.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.198+0000 7f8b7cff9640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-29T11:43:08.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.198+0000 7f8b7f7fe640 10 monclient: get_auth_request con 0x7f8b6c06dd00 auth_method 0 2024-03-29T11:43:08.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.199+0000 7f8b7cff9640 10 client.4468.objecter ms_handle_connect 0x7f8b6c06dd00 2024-03-29T11:43:08.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.208+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b700505c0 con 0x7f8b800a9d70 2024-03-29T11:43:08.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.208+0000 7f8b7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:08.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.208+0000 7f8b7cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:08.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.544+0000 7f8b667fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:08.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.545+0000 7f8b667fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:08.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.545+0000 7f8b667fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:08.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.545+0000 7f8b667fc640 1 -- 172.21.15.37:0/3498363524 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b60005330 con 0x7f8b800a9d70 2024-03-29T11:43:08.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.546+0000 7f8b7cff9640 1 -- 172.21.15.37:0/3498363524 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f8b70005d20 con 0x7f8b800a9d70 2024-03-29T11:43:08.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.546+0000 7f8b7cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:08.550 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:43:08.550 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:08.550 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.546+0000 7f8b7cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:08.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.549+0000 7f8b667fc640 10 librados: watch_flush enter 2024-03-29T11:43:08.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.549+0000 7f8b667fc640 10 librados: watch_flush exit 2024-03-29T11:43:08.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.549+0000 7f8b667fc640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-29T11:43:08.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.549+0000 7f8b667fc640 10 client.4468.objecter successfully canceled tick 2024-03-29T11:43:08.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.550+0000 7f8b667fc640 1 -- 172.21.15.37:0/3498363524 >> v1:172.21.15.146:6816/2899149139 conn(0x7f8b6c06dd00 legacy=0x7f8b6c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.550+0000 7f8b667fc640 10 monclient: shutdown 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.550+0000 7f8b667fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.550+0000 7f8b667fc640 1 -- 172.21.15.37:0/3498363524 >> v1:172.21.15.146:6789/0 conn(0x7f8b800a9d70 legacy=0x7f8b80138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.550+0000 7f8b667fc640 1 -- 172.21.15.37:0/3498363524 shutdown_connections 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.550+0000 7f8b667fc640 1 -- 172.21.15.37:0/3498363524 >> 172.21.15.37:0/3498363524 conn(0x7f8b80066a00 msgr2=0x7f8b800b6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.550+0000 7f8b667fc640 1 -- 172.21.15.37:0/3498363524 shutdown_connections 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.550+0000 7f8b667fc640 1 -- 172.21.15.37:0/3498363524 wait complete. 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:08.550+0000 7f8b667fc640 1 librados: shutdown 2024-03-29T11:43:08.570 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:15.378 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:15.533 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.530+0000 7f4606115640 10 monclient: get_monmap_and_config 2024-03-29T11:43:15.533 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.530+0000 7f4606115640 10 monclient: build_initial_monmap 2024-03-29T11:43:15.533 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.530+0000 7f4606115640 10 monclient: monmap: 2024-03-29T11:43:15.534 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:15.535 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:15.535 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:43:15.530945+0000 2024-03-29T11:43:15.535 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:43:15.530945+0000 2024-03-29T11:43:15.535 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:15.535 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:15.535 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:15.535 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:43:15.535 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:43:15.535 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:15.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.531+0000 7f4606115640 1 Processor -- start 2024-03-29T11:43:15.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.532+0000 7f4606115640 1 -- start start 2024-03-29T11:43:15.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.532+0000 7f4606115640 10 monclient: init 2024-03-29T11:43:15.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.533+0000 7f4606115640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:15.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.533+0000 7f4606115640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:43:15.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.533+0000 7f4606115640 10 monclient(hunting): picked mon.noname-c con 0x7f46000b4790 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:15.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.533+0000 7f4606115640 10 monclient(hunting): picked mon.noname-a con 0x7f46000aea90 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:15.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.533+0000 7f4606115640 10 monclient(hunting): picked mon.noname-b con 0x7f46000b39f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:15.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.533+0000 7f4606115640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46000b0c60 con 0x7f46000aea90 2024-03-29T11:43:15.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.533+0000 7f4606115640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46000af020 con 0x7f46000b4790 2024-03-29T11:43:15.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.533+0000 7f4606115640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460013fbd0 con 0x7f46000b39f0 2024-03-29T11:43:15.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.533+0000 7f4606115640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:15.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.533+0000 7f4606115640 10 monclient(hunting): authenticate will time out at 1374.982513s 2024-03-29T11:43:15.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.537+0000 7f45fffff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f46000b39f0 0x7f460013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:48511/0 (socket says 172.21.15.37:48511) 2024-03-29T11:43:15.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.537+0000 7f45fffff640 1 -- 172.21.15.37:0/2190394225 learned_addr learned my addr 172.21.15.37:0/2190394225 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:15.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 1 -- 172.21.15.37:0/2190394225 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567630758 0 0) 0x7f460013fbd0 con 0x7f46000b39f0 2024-03-29T11:43:15.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:15.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:15.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:15.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:43:15.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 1 -- 172.21.15.37:0/2190394225 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e8003520 con 0x7f46000b39f0 2024-03-29T11:43:15.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 1 -- 172.21.15.37:0/2190394225 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942503806 0 0) 0x7f46000af020 con 0x7f46000b4790 2024-03-29T11:43:15.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:15.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:15.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:15.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:43:15.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 1 -- 172.21.15.37:0/2190394225 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460013fbd0 con 0x7f46000b4790 2024-03-29T11:43:15.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 1 -- 172.21.15.37:0/2190394225 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067080295 0 0) 0x7f46000b0c60 con 0x7f46000aea90 2024-03-29T11:43:15.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:15.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:15.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:15.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:43:15.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 1 -- 172.21.15.37:0/2190394225 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46000af020 con 0x7f46000aea90 2024-03-29T11:43:15.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 1 -- 172.21.15.37:0/2190394225 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 49989123 0 0) 0x7f46000af020 con 0x7f46000aea90 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 1 -- 172.21.15.37:0/2190394225 >> v1:172.21.15.37:6790/0 conn(0x7f46000b39f0 legacy=0x7f460013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 1 -- 172.21.15.37:0/2190394225 >> v1:172.21.15.146:6789/0 conn(0x7f46000b4790 legacy=0x7f46000b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 1 monclient: found mon.noname-a 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 1 -- 172.21.15.37:0/2190394225 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460013fd60 con 0x7f46000aea90 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 1 -- 172.21.15.37:0/2190394225 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f45f00015b0 con 0x7f46000aea90 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.538+0000 7f45fe7fc640 10 monclient: dump: 2024-03-29T11:43:15.546 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:15.547 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:15.547 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:15.547 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:15.547 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:15.547 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:15.547 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:15.547 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:15.547 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:15.547 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:15.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.539+0000 7f4606115640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:43:15.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.539+0000 7f4606115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:15.550 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.539+0000 7f45fe7fc640 1 -- 172.21.15.37:0/2190394225 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45f0001710 con 0x7f46000aea90 2024-03-29T11:43:15.550 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.539+0000 7f45fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:15.550 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.539+0000 7f45fe7fc640 1 -- 172.21.15.37:0/2190394225 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f45f0001b20 con 0x7f46000aea90 2024-03-29T11:43:15.550 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.539+0000 7f45fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:15.550 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.539+0000 7f45fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:15.550 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.539+0000 7f45fe7fc640 10 monclient: dump: 2024-03-29T11:43:15.550 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:15.550 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:15.550 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:15.550 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:15.550 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:15.550 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.539+0000 7f4606115640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.539+0000 7f4606115640 4 set_mon_vals no callback set 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.540+0000 7f4606115640 10 monclient: shutdown 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.540+0000 7f4606115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.540+0000 7f4606115640 1 -- 172.21.15.37:0/2190394225 >> v1:172.21.15.37:6789/0 conn(0x7f46000aea90 legacy=0x7f46000af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.542+0000 7f4606115640 1 -- 172.21.15.37:0/2190394225 shutdown_connections 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.542+0000 7f4606115640 1 -- 172.21.15.37:0/2190394225 >> 172.21.15.37:0/2190394225 conn(0x7f4600066a00 msgr2=0x7f4600066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.543+0000 7f4606115640 1 -- 172.21.15.37:0/2190394225 shutdown_connections 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.543+0000 7f4606115640 1 -- 172.21.15.37:0/2190394225 wait complete. 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.544+0000 7f4606115640 10 monclient: build_initial_monmap 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.544+0000 7f4606115640 10 monclient: monmap: 2024-03-29T11:43:15.551 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.544+0000 7f4606115640 1 librados: starting msgr at 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.544+0000 7f4606115640 1 librados: starting objecter 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.544+0000 7f4606115640 1 Processor -- start 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.545+0000 7f4606115640 1 -- start start 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.545+0000 7f4606115640 1 librados: setting wanted keys 2024-03-29T11:43:15.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.545+0000 7f4606115640 1 librados: calling monclient init 2024-03-29T11:43:15.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.545+0000 7f4606115640 10 monclient: init 2024-03-29T11:43:15.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.545+0000 7f4606115640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:15.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.545+0000 7f4606115640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:15.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.548+0000 7f4606115640 10 monclient(hunting): picked mon.noname-a con 0x7f46000b39f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:15.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.548+0000 7f4606115640 10 monclient(hunting): picked mon.noname-c con 0x7f46000b4790 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:15.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.548+0000 7f4606115640 10 monclient(hunting): picked mon.noname-b con 0x7f4600071120 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:15.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.548+0000 7f4606115640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4600141340 con 0x7f46000b39f0 2024-03-29T11:43:15.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.548+0000 7f4606115640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460013ff70 con 0x7f4600071120 2024-03-29T11:43:15.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.548+0000 7f4606115640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46000ae530 con 0x7f46000b4790 2024-03-29T11:43:15.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.548+0000 7f4606115640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:15.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.548+0000 7f4606115640 10 monclient(hunting): authenticate will time out at 1374.997748s 2024-03-29T11:43:15.555 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.552+0000 7f45ff7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f46000b39f0 0x7f46001e7240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49943/0 (socket says 172.21.15.37:49943) 2024-03-29T11:43:15.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.552+0000 7f45ff7fe640 1 -- 172.21.15.37:0/1297197077 learned_addr learned my addr 172.21.15.37:0/1297197077 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:15.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.552+0000 7f45e7fff640 10 client.?.objecter ms_handle_connect 0x7f46000b39f0 2024-03-29T11:43:15.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.552+0000 7f45e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:15.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.552+0000 7f45e7fff640 10 client.?.objecter ms_handle_connect 0x7f4600071120 2024-03-29T11:43:15.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.552+0000 7f45e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:15.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.552+0000 7f45e7fff640 10 client.?.objecter ms_handle_connect 0x7f46000b4790 2024-03-29T11:43:15.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.552+0000 7f45e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:15.559 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723198776 0 0) 0x7f460013ff70 con 0x7f4600071120 2024-03-29T11:43:15.559 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:15.559 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:15.559 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:15.559 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:43:15.559 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45ec004eb0 con 0x7f4600071120 2024-03-29T11:43:15.559 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900994657 0 0) 0x7f4600141340 con 0x7f46000b39f0 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460013ff70 con 0x7f46000b39f0 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866352018 0 0) 0x7f46000ae530 con 0x7f46000b4790 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4600141340 con 0x7f46000b4790 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1637513336 0 0) 0x7f460013ff70 con 0x7f46000b39f0 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46000ae530 con 0x7f46000b39f0 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f45f4001810 con 0x7f46000b39f0 2024-03-29T11:43:15.560 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): dump: 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 591832883 0 0) 0x7f45ec004eb0 con 0x7f4600071120 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f460013ff70 con 0x7f4600071120 2024-03-29T11:43:15.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f45f8001cd0 con 0x7f4600071120 2024-03-29T11:43:15.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:15.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:15.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.553+0000 7f45e7fff640 10 monclient(hunting): dump: 2024-03-29T11:43:15.562 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:15.562 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:15.562 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:15.562 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:15.562 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:15.562 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:15.562 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:15.562 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:15.562 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:15.562 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:15.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3962473288 0 0) 0x7f46000ae530 con 0x7f46000b39f0 2024-03-29T11:43:15.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 >> v1:172.21.15.37:6790/0 conn(0x7f46000b4790 legacy=0x7f46001d6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 >> v1:172.21.15.146:6789/0 conn(0x7f4600071120 legacy=0x7f46001ef580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f45e7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:15.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f45e7fff640 1 monclient: found mon.a 2024-03-29T11:43:15.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f45e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f45e7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4600070550 con 0x7f46000b39f0 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f45e7fff640 10 monclient: _finish_auth 0 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f45e7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f45e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f4606115640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f4606115640 10 monclient: _renew_subs 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f4606115640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f4606115640 1 -- 172.21.15.37:0/1297197077 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4600140490 con 0x7f46000b39f0 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f4606115640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f4606115640 10 monclient: _renew_subs 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f4606115640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f4606115640 1 -- 172.21.15.37:0/1297197077 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f460013eb90 con 0x7f46000b39f0 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.554+0000 7f4606115640 1 librados: init done 2024-03-29T11:43:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.557+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45f4001d00 con 0x7f46000b39f0 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.557+0000 7f45e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.557+0000 7f45fd7fa640 4 set_mon_vals no callback set 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.557+0000 7f45e5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.558+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f45f40037b0 con 0x7f46000b39f0 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.558+0000 7f45e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.558+0000 7f45e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.558+0000 7f45e7fff640 10 monclient: dump: 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:15.567 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:15.568 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:15.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.558+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f45f40039d0 con 0x7f46000b39f0 2024-03-29T11:43:15.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.558+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f45f40849f0 con 0x7f46000b39f0 2024-03-29T11:43:15.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.558+0000 7f45e7fff640 10 client.4512.objecter ms_dispatch 0x7f4600000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:15.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.558+0000 7f45e7fff640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:15.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.558+0000 7f45e7fff640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:15.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.558+0000 7f45e7fff640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-29T11:43:15.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.559+0000 7f45e5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:15.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.559+0000 7f45e5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:43:15.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.559+0000 7f45e5ffb640 1 -- 172.21.15.37:0/1297197077 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e0005080 con 0x7f46000b39f0 2024-03-29T11:43:15.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.559+0000 7f45feffd640 10 monclient: get_auth_request con 0x7f45ec06e050 auth_method 0 2024-03-29T11:43:15.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.560+0000 7f45e7fff640 10 client.4512.objecter ms_handle_connect 0x7f45ec06e050 2024-03-29T11:43:15.571 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.568+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45f4050350 con 0x7f46000b39f0 2024-03-29T11:43:15.571 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.568+0000 7f45e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:15.571 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.568+0000 7f45e7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:15.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.904+0000 7f45e5ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:15.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.904+0000 7f45e5ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:15.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.904+0000 7f45e5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:43:15.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.904+0000 7f45e5ffb640 1 -- 172.21.15.37:0/1297197077 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45e0005330 con 0x7f46000b39f0 2024-03-29T11:43:15.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.906+0000 7f45e7fff640 1 -- 172.21.15.37:0/1297197077 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f45f4052280 con 0x7f46000b39f0 2024-03-29T11:43:15.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.906+0000 7f45e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:15.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.906+0000 7f45e7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:15.910 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:43:15.910 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:15.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.910+0000 7f45e5ffb640 10 librados: watch_flush enter 2024-03-29T11:43:15.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.910+0000 7f45e5ffb640 10 librados: watch_flush exit 2024-03-29T11:43:15.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.910+0000 7f45e5ffb640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-29T11:43:15.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.910+0000 7f45e5ffb640 10 client.4512.objecter successfully canceled tick 2024-03-29T11:43:15.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.911+0000 7f45e5ffb640 1 -- 172.21.15.37:0/1297197077 >> v1:172.21.15.146:6816/2899149139 conn(0x7f45ec06e050 legacy=0x7f45ec070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.911+0000 7f45e5ffb640 10 monclient: shutdown 2024-03-29T11:43:15.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.911+0000 7f45e5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:15.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.911+0000 7f45e5ffb640 1 -- 172.21.15.37:0/1297197077 >> v1:172.21.15.37:6789/0 conn(0x7f46000b39f0 legacy=0x7f46001e7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.911+0000 7f45e5ffb640 1 -- 172.21.15.37:0/1297197077 shutdown_connections 2024-03-29T11:43:15.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.911+0000 7f45e5ffb640 1 -- 172.21.15.37:0/1297197077 >> 172.21.15.37:0/1297197077 conn(0x7f4600066a00 msgr2=0x7f460013f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:15.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.911+0000 7f45e5ffb640 1 -- 172.21.15.37:0/1297197077 shutdown_connections 2024-03-29T11:43:15.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.911+0000 7f45e5ffb640 1 -- 172.21.15.37:0/1297197077 wait complete. 2024-03-29T11:43:15.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:15.916+0000 7f45e5ffb640 1 librados: shutdown 2024-03-29T11:43:15.937 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:22.744 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:22.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.894+0000 7f6fcf527640 10 monclient: get_monmap_and_config 2024-03-29T11:43:22.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.894+0000 7f6fcf527640 10 monclient: build_initial_monmap 2024-03-29T11:43:22.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.894+0000 7f6fcf527640 10 monclient: monmap: 2024-03-29T11:43:22.898 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:22.898 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:22.898 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:43:22.895600+0000 2024-03-29T11:43:22.899 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:43:22.895600+0000 2024-03-29T11:43:22.899 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:22.899 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:22.899 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:22.899 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:43:22.899 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:43:22.899 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:22.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.897+0000 7f6fcf527640 1 Processor -- start 2024-03-29T11:43:22.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.897+0000 7f6fcf527640 1 -- start start 2024-03-29T11:43:22.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.897+0000 7f6fcf527640 10 monclient: init 2024-03-29T11:43:22.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.899+0000 7f6fcf527640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:22.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.899+0000 7f6fcf527640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:22.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.899+0000 7f6fcf527640 10 monclient(hunting): picked mon.noname-b con 0x7f6fc0006800 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:22.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.899+0000 7f6fcf527640 10 monclient(hunting): picked mon.noname-c con 0x7f6fc0006be0 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:22.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.899+0000 7f6fcf527640 10 monclient(hunting): picked mon.noname-a con 0x7f6fc00c95e0 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:22.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.899+0000 7f6fcf527640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc00c5570 con 0x7f6fc00c95e0 2024-03-29T11:43:22.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.899+0000 7f6fcf527640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc002e450 con 0x7f6fc0006be0 2024-03-29T11:43:22.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.899+0000 7f6fcf527640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc00c31c0 con 0x7f6fc0006800 2024-03-29T11:43:22.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.899+0000 7f6fcf527640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:22.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.899+0000 7f6fcf527640 10 monclient(hunting): authenticate will time out at 1382.348116s 2024-03-29T11:43:22.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.905+0000 7f6fce525640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f6fc0006800 0x7f6fc00c2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:35473/0 (socket says 172.21.15.37:35473) 2024-03-29T11:43:22.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.905+0000 7f6fce525640 1 -- 172.21.15.37:0/4223949826 learned_addr learned my addr 172.21.15.37:0/4223949826 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:22.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.905+0000 7f6fcd523640 1 -- 172.21.15.37:0/4223949826 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195334131 0 0) 0x7f6fc00c5570 con 0x7f6fc00c95e0 2024-03-29T11:43:22.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.905+0000 7f6fcd523640 10 monclient(hunting): _negotiate 2024-03-29T11:43:22.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.905+0000 7f6fcd523640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:22.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.905+0000 7f6fcd523640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:22.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.905+0000 7f6fcd523640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:43:22.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.905+0000 7f6fcd523640 1 -- 172.21.15.37:0/4223949826 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb4003520 con 0x7f6fc00c95e0 2024-03-29T11:43:22.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 1 -- 172.21.15.37:0/4223949826 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2783551644 0 0) 0x7f6fb4003520 con 0x7f6fc00c95e0 2024-03-29T11:43:22.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 1 -- 172.21.15.37:0/4223949826 >> v1:172.21.15.37:6790/0 conn(0x7f6fc0006800 legacy=0x7f6fc00c2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:22.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 1 -- 172.21.15.37:0/4223949826 >> v1:172.21.15.146:6789/0 conn(0x7f6fc0006be0 legacy=0x7f6fc00c8ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:43:22.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 10 monclient: _finish_hunting 0 2024-03-29T11:43:22.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 1 monclient: found mon.noname-a 2024-03-29T11:43:22.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:22.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 1 -- 172.21.15.37:0/4223949826 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc002cde0 con 0x7f6fc00c95e0 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 10 monclient: _finish_auth 0 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 10 monclient: _check_auth_tickets 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 1 -- 172.21.15.37:0/4223949826 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6fc804f730 con 0x7f6fc00c95e0 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 10 monclient: dump: 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:22.913 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcf527640 5 monclient: authenticate success, global_id 4515 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcf527640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 1 -- 172.21.15.37:0/4223949826 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fc8075f10 con 0x7f6fc00c95e0 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 1 -- 172.21.15.37:0/4223949826 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6fc80758b0 con 0x7f6fc00c95e0 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcd523640 10 monclient: dump: 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:22.914 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:22.915 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.915 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:22.915 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:22.915 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:22.915 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:22.915 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:22.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcf527640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:22.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.906+0000 7f6fcf527640 4 set_mon_vals no callback set 2024-03-29T11:43:22.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.907+0000 7f6fcf527640 10 monclient: shutdown 2024-03-29T11:43:22.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.907+0000 7f6fcf527640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:22.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.907+0000 7f6fcf527640 1 -- 172.21.15.37:0/4223949826 >> v1:172.21.15.37:6789/0 conn(0x7f6fc00c95e0 legacy=0x7f6fc00d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:22.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.908+0000 7f6fcf527640 1 -- 172.21.15.37:0/4223949826 shutdown_connections 2024-03-29T11:43:22.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.908+0000 7f6fcf527640 1 -- 172.21.15.37:0/4223949826 >> 172.21.15.37:0/4223949826 conn(0x7f6fc00058f0 msgr2=0x7f6fc0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:22.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.910+0000 7f6fcf527640 1 -- 172.21.15.37:0/4223949826 shutdown_connections 2024-03-29T11:43:22.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.910+0000 7f6fcf527640 1 -- 172.21.15.37:0/4223949826 wait complete. 2024-03-29T11:43:22.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.911+0000 7f6fcf527640 10 monclient: build_initial_monmap 2024-03-29T11:43:22.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.911+0000 7f6fcf527640 10 monclient: monmap: 2024-03-29T11:43:22.916 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:22.916 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:22.916 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:43:22.916 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:43:22.916 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:22.916 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:22.917 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:22.917 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:43:22.917 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:43:22.917 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:22.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.911+0000 7f6fcf527640 1 librados: starting msgr at 2024-03-29T11:43:22.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.911+0000 7f6fcf527640 1 librados: starting objecter 2024-03-29T11:43:22.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.911+0000 7f6fcf527640 1 Processor -- start 2024-03-29T11:43:22.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.911+0000 7f6fcf527640 1 -- start start 2024-03-29T11:43:22.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.911+0000 7f6fcf527640 1 librados: setting wanted keys 2024-03-29T11:43:22.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.911+0000 7f6fcf527640 1 librados: calling monclient init 2024-03-29T11:43:22.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.911+0000 7f6fcf527640 10 monclient: init 2024-03-29T11:43:22.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.912+0000 7f6fcf527640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:22.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.912+0000 7f6fcf527640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:22.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.912+0000 7f6fcf527640 10 monclient(hunting): picked mon.noname-a con 0x7f6fc0006800 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:22.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.912+0000 7f6fcf527640 10 monclient(hunting): picked mon.noname-b con 0x7f6fc00c95e0 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:22.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.912+0000 7f6fcf527640 10 monclient(hunting): picked mon.noname-c con 0x7f6fc00bff80 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:22.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.912+0000 7f6fcf527640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc00d4140 con 0x7f6fc0006800 2024-03-29T11:43:22.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.912+0000 7f6fcf527640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc00c32b0 con 0x7f6fc00c95e0 2024-03-29T11:43:22.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.912+0000 7f6fcf527640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc00d4390 con 0x7f6fc00bff80 2024-03-29T11:43:22.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.912+0000 7f6fcf527640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:22.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.912+0000 7f6fcf527640 10 monclient(hunting): authenticate will time out at 1382.361637s 2024-03-29T11:43:22.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.917+0000 7f6fced26640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f6fc00bff80 0x7f6fc0185b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:35559/0 (socket says 172.21.15.37:35559) 2024-03-29T11:43:22.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.917+0000 7f6fced26640 1 -- 172.21.15.37:0/2026298079 learned_addr learned my addr 172.21.15.37:0/2026298079 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:22.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.917+0000 7f6fbeffd640 10 client.?.objecter ms_handle_connect 0x7f6fc00bff80 2024-03-29T11:43:22.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.917+0000 7f6fbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:22.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.917+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539798241 0 0) 0x7f6fc00d4390 con 0x7f6fc00bff80 2024-03-29T11:43:22.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.917+0000 7f6fbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:22.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.917+0000 7f6fbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:22.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.917+0000 7f6fbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:22.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.917+0000 7f6fbeffd640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:43:22.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.917+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fac004eb0 con 0x7f6fc00bff80 2024-03-29T11:43:22.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.918+0000 7f6fbeffd640 10 client.?.objecter ms_handle_connect 0x7f6fc0006800 2024-03-29T11:43:22.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.918+0000 7f6fbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:22.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1188393941 0 0) 0x7f6fac004eb0 con 0x7f6fc00bff80 2024-03-29T11:43:22.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fc00d4390 con 0x7f6fc00bff80 2024-03-29T11:43:22.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6fc804f400 con 0x7f6fc00bff80 2024-03-29T11:43:22.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:22.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient(hunting): dump: 2024-03-29T11:43:22.928 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:22.928 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850573976 0 0) 0x7f6fc00d4140 con 0x7f6fc0006800 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:43:22.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fac005640 con 0x7f6fc0006800 2024-03-29T11:43:22.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 969580259 0 0) 0x7f6fc00d4390 con 0x7f6fc00bff80 2024-03-29T11:43:22.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 >> v1:172.21.15.146:6789/0 conn(0x7f6fc00c95e0 legacy=0x7f6fc00bc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:22.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 >> v1:172.21.15.37:6789/0 conn(0x7f6fc0006800 legacy=0x7f6fc00c2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:22.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:22.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 1 monclient: found mon.c 2024-03-29T11:43:22.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:22.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:22.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc00bba30 con 0x7f6fc00bff80 2024-03-29T11:43:22.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:22.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:22.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:22.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fcf527640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:43:22.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fcf527640 10 monclient: _renew_subs 2024-03-29T11:43:22.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fcf527640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:22.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fcf527640 1 -- 172.21.15.37:0/2026298079 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc002ffb0 con 0x7f6fc00bff80 2024-03-29T11:43:22.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fcf527640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:22.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fcf527640 10 monclient: _renew_subs 2024-03-29T11:43:22.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fcf527640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:22.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fcf527640 1 -- 172.21.15.37:0/2026298079 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc0032bc0 con 0x7f6fc00bff80 2024-03-29T11:43:22.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fcf527640 1 librados: init done 2024-03-29T11:43:22.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fc804fdd0 con 0x7f6fc00bff80 2024-03-29T11:43:22.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:22.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6fc804f400 con 0x7f6fc00bff80 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.919+0000 7f6fbeffd640 10 monclient: dump: 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.921+0000 7f6fcf527640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:22.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.921+0000 7f6fcf527640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:22.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.921+0000 7f6fcf527640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:22.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.921+0000 7f6fcf527640 1 -- 172.21.15.37:0/2026298079 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc0004cd0 con 0x7f6fc00bff80 2024-03-29T11:43:22.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.921+0000 7f6fccd22640 4 set_mon_vals no callback set 2024-03-29T11:43:22.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.921+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f6fc8067bf0 con 0x7f6fc00bff80 2024-03-29T11:43:22.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.922+0000 7f6fce525640 10 monclient: get_auth_request con 0x7f6fac06e1d0 auth_method 0 2024-03-29T11:43:22.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.922+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f6fc80cefd0 con 0x7f6fc00bff80 2024-03-29T11:43:22.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.922+0000 7f6fbeffd640 10 client.4574.objecter ms_dispatch 0x7f6fc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:22.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.922+0000 7f6fbeffd640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:22.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.922+0000 7f6fbeffd640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:22.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.922+0000 7f6fbeffd640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-29T11:43:22.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.923+0000 7f6fbeffd640 10 client.4574.objecter ms_handle_connect 0x7f6fac06e1d0 2024-03-29T11:43:22.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.931+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6fc80cb270 con 0x7f6fc00bff80 2024-03-29T11:43:22.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.931+0000 7f6fbeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:22.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:22.931+0000 7f6fbeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:23.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.317+0000 7f6fcf527640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:23.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.317+0000 7f6fcf527640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:23.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.317+0000 7f6fcf527640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:23.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.317+0000 7f6fcf527640 1 -- 172.21.15.37:0/2026298079 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fc00352b0 con 0x7f6fc00bff80 2024-03-29T11:43:23.322 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.318+0000 7f6fbeffd640 1 -- 172.21.15.37:0/2026298079 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f6fc80cb270 con 0x7f6fc00bff80 2024-03-29T11:43:23.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.318+0000 7f6fbeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:23.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.318+0000 7f6fbeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:23.327 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:43:23.327 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:23.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.322+0000 7f6fcf527640 10 librados: watch_flush enter 2024-03-29T11:43:23.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.322+0000 7f6fcf527640 10 librados: watch_flush exit 2024-03-29T11:43:23.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.322+0000 7f6fcf527640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-29T11:43:23.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.322+0000 7f6fcf527640 10 client.4574.objecter successfully canceled tick 2024-03-29T11:43:23.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.322+0000 7f6fcf527640 1 -- 172.21.15.37:0/2026298079 >> v1:172.21.15.146:6816/2899149139 conn(0x7f6fac06e1d0 legacy=0x7f6fac070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:23.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.322+0000 7f6fcf527640 10 monclient: shutdown 2024-03-29T11:43:23.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.322+0000 7f6fcf527640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:23.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.322+0000 7f6fcf527640 1 -- 172.21.15.37:0/2026298079 >> v1:172.21.15.37:6790/0 conn(0x7f6fc00bff80 legacy=0x7f6fc0185b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:23.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.322+0000 7f6fcf527640 1 -- 172.21.15.37:0/2026298079 shutdown_connections 2024-03-29T11:43:23.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.322+0000 7f6fcf527640 1 -- 172.21.15.37:0/2026298079 >> 172.21.15.37:0/2026298079 conn(0x7f6fc00058f0 msgr2=0x7f6fc00c6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:23.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.323+0000 7f6fcf527640 1 -- 172.21.15.37:0/2026298079 shutdown_connections 2024-03-29T11:43:23.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.323+0000 7f6fcf527640 1 -- 172.21.15.37:0/2026298079 wait complete. 2024-03-29T11:43:23.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:23.323+0000 7f6fcf527640 1 librados: shutdown 2024-03-29T11:43:23.341 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:30.148 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:30.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.293+0000 7fda8c016640 10 monclient: get_monmap_and_config 2024-03-29T11:43:30.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.293+0000 7fda8c016640 10 monclient: build_initial_monmap 2024-03-29T11:43:30.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.293+0000 7fda8c016640 10 monclient: monmap: 2024-03-29T11:43:30.298 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:30.298 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:30.298 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:43:30.293941+0000 2024-03-29T11:43:30.298 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:43:30.293941+0000 2024-03-29T11:43:30.298 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:30.298 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:30.298 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:30.298 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:43:30.298 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:43:30.298 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:30.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.294+0000 7fda8c016640 1 Processor -- start 2024-03-29T11:43:30.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.295+0000 7fda8c016640 1 -- start start 2024-03-29T11:43:30.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.295+0000 7fda8c016640 10 monclient: init 2024-03-29T11:43:30.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.296+0000 7fda8c016640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:30.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.296+0000 7fda8c016640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:30.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.296+0000 7fda8c016640 10 monclient(hunting): picked mon.noname-a con 0x7fda840a9ac0 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:30.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.296+0000 7fda8c016640 10 monclient(hunting): picked mon.noname-b con 0x7fda8413ee90 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:30.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.296+0000 7fda8c016640 10 monclient(hunting): picked mon.noname-c con 0x7fda840af900 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:30.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.296+0000 7fda8c016640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda840afd10 con 0x7fda840a9ac0 2024-03-29T11:43:30.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.296+0000 7fda8c016640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda840af6f0 con 0x7fda840af900 2024-03-29T11:43:30.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.296+0000 7fda8c016640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda840aac80 con 0x7fda8413ee90 2024-03-29T11:43:30.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.296+0000 7fda8c016640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:30.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.296+0000 7fda8c016640 10 monclient(hunting): authenticate will time out at 1389.745081s 2024-03-29T11:43:30.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.299+0000 7fda89db2640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fda840a9ac0 0x7fda840adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48645/0 (socket says 172.21.15.37:48645) 2024-03-29T11:43:30.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.299+0000 7fda89db2640 1 -- 172.21.15.37:0/708988682 learned_addr learned my addr 172.21.15.37:0/708988682 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:30.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 1 -- 172.21.15.37:0/708988682 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098576406 0 0) 0x7fda840aac80 con 0x7fda8413ee90 2024-03-29T11:43:30.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient(hunting): _negotiate 2024-03-29T11:43:30.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:30.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:30.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:43:30.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 1 -- 172.21.15.37:0/708988682 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda70003520 con 0x7fda8413ee90 2024-03-29T11:43:30.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 1 -- 172.21.15.37:0/708988682 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888066585 0 0) 0x7fda840afd10 con 0x7fda840a9ac0 2024-03-29T11:43:30.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient(hunting): _negotiate 2024-03-29T11:43:30.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:30.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:30.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:43:30.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 1 -- 172.21.15.37:0/708988682 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda840aac80 con 0x7fda840a9ac0 2024-03-29T11:43:30.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 1 -- 172.21.15.37:0/708988682 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4133239427 0 0) 0x7fda70003520 con 0x7fda8413ee90 2024-03-29T11:43:30.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 1 -- 172.21.15.37:0/708988682 >> v1:172.21.15.146:6789/0 conn(0x7fda840af900 legacy=0x7fda84149b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:43:30.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 1 -- 172.21.15.37:0/708988682 >> v1:172.21.15.37:6789/0 conn(0x7fda840a9ac0 legacy=0x7fda840adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient: _finish_hunting 0 2024-03-29T11:43:30.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 1 monclient: found mon.noname-b 2024-03-29T11:43:30.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:30.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:43:30.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 1 -- 172.21.15.37:0/708988682 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda840aaf60 con 0x7fda8413ee90 2024-03-29T11:43:30.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient: _finish_auth 0 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient: _check_auth_tickets 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 1 -- 172.21.15.37:0/708988682 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fda74001690 con 0x7fda8413ee90 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.301+0000 7fda88db0640 10 monclient: dump: 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:30.307 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.302+0000 7fda88db0640 1 -- 172.21.15.37:0/708988682 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda740017f0 con 0x7fda8413ee90 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.302+0000 7fda88db0640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.302+0000 7fda88db0640 1 -- 172.21.15.37:0/708988682 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fda74001c00 con 0x7fda8413ee90 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.302+0000 7fda88db0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.302+0000 7fda88db0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.302+0000 7fda88db0640 10 monclient: dump: 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:30.308 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:30.309 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:30.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.302+0000 7fda8c016640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:43:30.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.302+0000 7fda8c016640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:30.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.302+0000 7fda8c016640 4 set_mon_vals no callback set 2024-03-29T11:43:30.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.303+0000 7fda8c016640 10 monclient: shutdown 2024-03-29T11:43:30.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.303+0000 7fda8c016640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:30.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.303+0000 7fda8c016640 1 -- 172.21.15.37:0/708988682 >> v1:172.21.15.37:6790/0 conn(0x7fda8413ee90 legacy=0x7fda8413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.303+0000 7fda8c016640 1 -- 172.21.15.37:0/708988682 shutdown_connections 2024-03-29T11:43:30.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.303+0000 7fda8c016640 1 -- 172.21.15.37:0/708988682 >> 172.21.15.37:0/708988682 conn(0x7fda840b35e0 msgr2=0x7fda840b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:30.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.307+0000 7fda8c016640 1 -- 172.21.15.37:0/708988682 shutdown_connections 2024-03-29T11:43:30.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.307+0000 7fda8c016640 1 -- 172.21.15.37:0/708988682 wait complete. 2024-03-29T11:43:30.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.308+0000 7fda8c016640 10 monclient: build_initial_monmap 2024-03-29T11:43:30.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.308+0000 7fda8c016640 10 monclient: monmap: 2024-03-29T11:43:30.317 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:30.317 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:30.317 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:43:30.317 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:43:30.317 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:30.317 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:30.317 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:30.318 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:43:30.318 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:43:30.318 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:30.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.308+0000 7fda8c016640 1 librados: starting msgr at 2024-03-29T11:43:30.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.308+0000 7fda8c016640 1 librados: starting objecter 2024-03-29T11:43:30.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.308+0000 7fda8c016640 1 Processor -- start 2024-03-29T11:43:30.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.308+0000 7fda8c016640 1 -- start start 2024-03-29T11:43:30.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.308+0000 7fda8c016640 1 librados: setting wanted keys 2024-03-29T11:43:30.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.308+0000 7fda8c016640 1 librados: calling monclient init 2024-03-29T11:43:30.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.308+0000 7fda8c016640 10 monclient: init 2024-03-29T11:43:30.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.309+0000 7fda8c016640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:30.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.309+0000 7fda8c016640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:30.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.309+0000 7fda8c016640 10 monclient(hunting): picked mon.noname-a con 0x7fda840a9ac0 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:30.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.309+0000 7fda8c016640 10 monclient(hunting): picked mon.noname-b con 0x7fda840af900 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:30.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.309+0000 7fda8c016640 10 monclient(hunting): picked mon.noname-c con 0x7fda8413ee90 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:30.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.309+0000 7fda8c016640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda841421d0 con 0x7fda840a9ac0 2024-03-29T11:43:30.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.309+0000 7fda8c016640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda84142440 con 0x7fda840af900 2024-03-29T11:43:30.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.309+0000 7fda8c016640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda841f5c40 con 0x7fda8413ee90 2024-03-29T11:43:30.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.309+0000 7fda8c016640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:30.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.309+0000 7fda8c016640 10 monclient(hunting): authenticate will time out at 1389.758712s 2024-03-29T11:43:30.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.314+0000 7fda89db2640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fda840a9ac0 0x7fda841f5390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48837/0 (socket says 172.21.15.37:48837) 2024-03-29T11:43:30.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.314+0000 7fda89db2640 1 -- 172.21.15.37:0/4164376107 learned_addr learned my addr 172.21.15.37:0/4164376107 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:30.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 10 client.?.objecter ms_handle_connect 0x7fda8413ee90 2024-03-29T11:43:30.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:30.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 10 client.?.objecter ms_handle_connect 0x7fda840a9ac0 2024-03-29T11:43:30.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:30.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189687262 0 0) 0x7fda841f5c40 con 0x7fda8413ee90 2024-03-29T11:43:30.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:30.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:30.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:30.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:43:30.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda68004eb0 con 0x7fda8413ee90 2024-03-29T11:43:30.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 10 client.?.objecter ms_handle_connect 0x7fda840af900 2024-03-29T11:43:30.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:30.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950905403 0 0) 0x7fda841421d0 con 0x7fda840a9ac0 2024-03-29T11:43:30.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:30.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:30.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:30.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:43:30.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.316+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda841f5c40 con 0x7fda840a9ac0 2024-03-29T11:43:30.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.317+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347015736 0 0) 0x7fda84142440 con 0x7fda840af900 2024-03-29T11:43:30.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.317+0000 7fda7a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:30.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.317+0000 7fda7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:30.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.317+0000 7fda7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:30.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.317+0000 7fda7a7fc640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:43:30.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.317+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda841421d0 con 0x7fda840af900 2024-03-29T11:43:30.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.317+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 671668997 0 0) 0x7fda68004eb0 con 0x7fda8413ee90 2024-03-29T11:43:30.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.317+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda84142440 con 0x7fda8413ee90 2024-03-29T11:43:30.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.317+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fda7c001af0 con 0x7fda8413ee90 2024-03-29T11:43:30.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.317+0000 7fda7a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.317+0000 7fda7a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:30.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.317+0000 7fda7a7fc640 10 monclient(hunting): dump: 2024-03-29T11:43:30.330 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:30.330 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:30.330 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:30.330 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:30.331 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.331 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:30.331 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:30.331 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:30.331 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:30.331 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:30.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.317+0000 7fda7a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:30.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.318+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1821425396 0 0) 0x7fda841f5c40 con 0x7fda840a9ac0 2024-03-29T11:43:30.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.318+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda68004eb0 con 0x7fda840a9ac0 2024-03-29T11:43:30.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.318+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fda800018b0 con 0x7fda840a9ac0 2024-03-29T11:43:30.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.318+0000 7fda7a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.318+0000 7fda7a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:30.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.318+0000 7fda7a7fc640 10 monclient(hunting): dump: 2024-03-29T11:43:30.332 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:30.332 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:30.332 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:30.333 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:30.333 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.333 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:30.333 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:30.333 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:30.333 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:30.333 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:30.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 730694924 0 0) 0x7fda841421d0 con 0x7fda840af900 2024-03-29T11:43:30.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda841f5c40 con 0x7fda840af900 2024-03-29T11:43:30.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fda74001b00 con 0x7fda840af900 2024-03-29T11:43:30.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:30.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 10 monclient(hunting): dump: 2024-03-29T11:43:30.335 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:30.335 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:30.335 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:30.335 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:30.335 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.336 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:30.336 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:30.336 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:30.336 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:30.336 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:30.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2623294749 0 0) 0x7fda84142440 con 0x7fda8413ee90 2024-03-29T11:43:30.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 >> v1:172.21.15.146:6789/0 conn(0x7fda840af900 legacy=0x7fda84140270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 >> v1:172.21.15.37:6789/0 conn(0x7fda840a9ac0 legacy=0x7fda841f5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:30.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 1 monclient: found mon.c 2024-03-29T11:43:30.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:30.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:30.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda84071f90 con 0x7fda8413ee90 2024-03-29T11:43:30.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:30.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:30.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.320+0000 7fda7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:30.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda7c002000 con 0x7fda8413ee90 2024-03-29T11:43:30.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:30.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fda7c003bb0 con 0x7fda8413ee90 2024-03-29T11:43:30.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:30.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda7a7fc640 10 monclient: dump: 2024-03-29T11:43:30.337 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:30.338 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:30.338 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:30.338 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:30.338 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.338 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:30.338 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:30.338 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:30.338 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:30.338 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:30.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda8c016640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:43:30.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda8c016640 10 monclient: _renew_subs 2024-03-29T11:43:30.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda8c016640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:30.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda8c016640 1 -- 172.21.15.37:0/4164376107 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda84072240 con 0x7fda8413ee90 2024-03-29T11:43:30.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda8c016640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:30.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda8c016640 10 monclient: _renew_subs 2024-03-29T11:43:30.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda8c016640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:30.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda8c016640 1 -- 172.21.15.37:0/4164376107 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda84200b00 con 0x7fda8413ee90 2024-03-29T11:43:30.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.321+0000 7fda8c016640 1 librados: init done 2024-03-29T11:43:30.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.322+0000 7fda7bfff640 4 set_mon_vals no callback set 2024-03-29T11:43:30.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.324+0000 7fda8c016640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:30.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.324+0000 7fda8c016640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:30.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.324+0000 7fda8c016640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:30.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.324+0000 7fda8c016640 1 -- 172.21.15.37:0/4164376107 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6c005080 con 0x7fda8413ee90 2024-03-29T11:43:30.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.325+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fda7c019b80 con 0x7fda8413ee90 2024-03-29T11:43:30.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.326+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fda7c083aa0 con 0x7fda8413ee90 2024-03-29T11:43:30.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.326+0000 7fda7a7fc640 10 client.4580.objecter ms_dispatch 0x7fda84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:30.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.326+0000 7fda7a7fc640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:30.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.326+0000 7fda7a7fc640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:30.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.326+0000 7fda7a7fc640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T11:43:30.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.329+0000 7fda89db2640 10 monclient: get_auth_request con 0x7fda6806e1f0 auth_method 0 2024-03-29T11:43:30.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.329+0000 7fda7a7fc640 10 client.4580.objecter ms_handle_connect 0x7fda6806e1f0 2024-03-29T11:43:30.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.334+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda7c04f1b0 con 0x7fda8413ee90 2024-03-29T11:43:30.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.334+0000 7fda7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:30.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.334+0000 7fda7a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:30.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.667+0000 7fda8c016640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:30.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.667+0000 7fda8c016640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:30.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.667+0000 7fda8c016640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:30.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.667+0000 7fda8c016640 1 -- 172.21.15.37:0/4164376107 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda6c005330 con 0x7fda8413ee90 2024-03-29T11:43:30.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.668+0000 7fda7a7fc640 1 -- 172.21.15.37:0/4164376107 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fda7c0512b0 con 0x7fda8413ee90 2024-03-29T11:43:30.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.668+0000 7fda7a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:30.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.668+0000 7fda7a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:30.673 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:43:30.674 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:30.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.673+0000 7fda8c016640 10 librados: watch_flush enter 2024-03-29T11:43:30.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.673+0000 7fda8c016640 10 librados: watch_flush exit 2024-03-29T11:43:30.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.673+0000 7fda8c016640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T11:43:30.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.673+0000 7fda8c016640 10 client.4580.objecter successfully canceled tick 2024-03-29T11:43:30.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.673+0000 7fda8c016640 1 -- 172.21.15.37:0/4164376107 >> v1:172.21.15.146:6816/2899149139 conn(0x7fda6806e1f0 legacy=0x7fda680706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.673+0000 7fda8c016640 10 monclient: shutdown 2024-03-29T11:43:30.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.673+0000 7fda8c016640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:30.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.673+0000 7fda8c016640 1 -- 172.21.15.37:0/4164376107 >> v1:172.21.15.37:6790/0 conn(0x7fda8413ee90 legacy=0x7fda84140980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.674+0000 7fda8c016640 1 -- 172.21.15.37:0/4164376107 shutdown_connections 2024-03-29T11:43:30.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.674+0000 7fda8c016640 1 -- 172.21.15.37:0/4164376107 >> 172.21.15.37:0/4164376107 conn(0x7fda840b35e0 msgr2=0x7fda840adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:30.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.674+0000 7fda8c016640 1 -- 172.21.15.37:0/4164376107 shutdown_connections 2024-03-29T11:43:30.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.674+0000 7fda8c016640 1 -- 172.21.15.37:0/4164376107 wait complete. 2024-03-29T11:43:30.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:30.674+0000 7fda8c016640 1 librados: shutdown 2024-03-29T11:43:30.703 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:37.510 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:37.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.668+0000 7f4c4326a640 10 monclient: get_monmap_and_config 2024-03-29T11:43:37.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.668+0000 7f4c4326a640 10 monclient: build_initial_monmap 2024-03-29T11:43:37.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.668+0000 7f4c4326a640 10 monclient: monmap: 2024-03-29T11:43:37.673 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:37.673 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:37.673 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:43:37.669566+0000 2024-03-29T11:43:37.673 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:43:37.669566+0000 2024-03-29T11:43:37.673 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:37.673 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:37.674 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:37.674 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:43:37.674 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:43:37.674 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:37.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.670+0000 7f4c4326a640 1 Processor -- start 2024-03-29T11:43:37.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.671+0000 7f4c4326a640 1 -- start start 2024-03-29T11:43:37.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.671+0000 7f4c4326a640 10 monclient: init 2024-03-29T11:43:37.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.671+0000 7f4c4326a640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:37.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.671+0000 7f4c4326a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:37.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.671+0000 7f4c4326a640 10 monclient(hunting): picked mon.noname-b con 0x7f4c3c0a9ac0 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:37.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.671+0000 7f4c4326a640 10 monclient(hunting): picked mon.noname-a con 0x7f4c3c13ee90 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:37.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.671+0000 7f4c4326a640 10 monclient(hunting): picked mon.noname-c con 0x7f4c3c0af900 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:37.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.671+0000 7f4c4326a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3c0afd10 con 0x7f4c3c13ee90 2024-03-29T11:43:37.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.671+0000 7f4c4326a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3c0af6f0 con 0x7f4c3c0af900 2024-03-29T11:43:37.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.671+0000 7f4c4326a640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3c0aac80 con 0x7f4c3c0a9ac0 2024-03-29T11:43:37.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.671+0000 7f4c4326a640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:37.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.671+0000 7f4c4326a640 10 monclient(hunting): authenticate will time out at 1397.120918s 2024-03-29T11:43:37.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.673+0000 7f4c41006640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f4c3c0a9ac0 0x7f4c3c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:51786/0 (socket says 172.21.15.37:51786) 2024-03-29T11:43:37.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.673+0000 7f4c41006640 1 -- 172.21.15.37:0/678452641 learned_addr learned my addr 172.21.15.37:0/678452641 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:37.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.678+0000 7f4c33fff640 1 -- 172.21.15.37:0/678452641 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890799883 0 0) 0x7f4c3c0aac80 con 0x7f4c3c0a9ac0 2024-03-29T11:43:37.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.678+0000 7f4c33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:37.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.678+0000 7f4c33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:37.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.678+0000 7f4c33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:37.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.678+0000 7f4c33fff640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:43:37.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.678+0000 7f4c33fff640 1 -- 172.21.15.37:0/678452641 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c28003520 con 0x7f4c3c0a9ac0 2024-03-29T11:43:37.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 1 -- 172.21.15.37:0/678452641 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305464335 0 0) 0x7f4c3c0af6f0 con 0x7f4c3c0af900 2024-03-29T11:43:37.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:37.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:37.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:37.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:43:37.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 1 -- 172.21.15.37:0/678452641 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c28003840 con 0x7f4c3c0af900 2024-03-29T11:43:37.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 1 -- 172.21.15.37:0/678452641 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3299535515 0 0) 0x7f4c28003520 con 0x7f4c3c0a9ac0 2024-03-29T11:43:37.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 1 -- 172.21.15.37:0/678452641 >> v1:172.21.15.146:6789/0 conn(0x7f4c3c0af900 legacy=0x7f4c3c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 1 -- 172.21.15.37:0/678452641 >> v1:172.21.15.37:6789/0 conn(0x7f4c3c13ee90 legacy=0x7f4c3c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 1 monclient: found mon.noname-b 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 1 -- 172.21.15.37:0/678452641 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c0aaf60 con 0x7f4c3c0a9ac0 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 10 monclient: _finish_auth 0 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c4326a640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c4326a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 1 -- 172.21.15.37:0/678452641 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4c2c004c90 con 0x7f4c3c0a9ac0 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.679+0000 7f4c33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.680+0000 7f4c33fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.680+0000 7f4c33fff640 10 monclient: dump: 2024-03-29T11:43:37.684 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:37.685 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:37.685 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:37.685 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:37.685 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:37.685 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:37.685 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:37.685 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:37.685 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:37.685 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:37.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.680+0000 7f4c4326a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:37.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.680+0000 7f4c33fff640 1 -- 172.21.15.37:0/678452641 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c2c004e70 con 0x7f4c3c0a9ac0 2024-03-29T11:43:37.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.680+0000 7f4c33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:37.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.680+0000 7f4c4326a640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:37.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.680+0000 7f4c4326a640 4 set_mon_vals no callback set 2024-03-29T11:43:37.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.681+0000 7f4c4326a640 10 monclient: shutdown 2024-03-29T11:43:37.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.681+0000 7f4c4326a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:37.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.681+0000 7f4c4326a640 1 -- 172.21.15.37:0/678452641 >> v1:172.21.15.37:6790/0 conn(0x7f4c3c0a9ac0 legacy=0x7f4c3c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.681+0000 7f4c33fff640 1 -- 172.21.15.37:0/678452641 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4c2c004c90 con 0x7f4c3c0a9ac0 2024-03-29T11:43:37.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.681+0000 7f4c33fff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:43:37.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.681+0000 7f4c4326a640 1 -- 172.21.15.37:0/678452641 shutdown_connections 2024-03-29T11:43:37.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.681+0000 7f4c4326a640 1 -- 172.21.15.37:0/678452641 >> 172.21.15.37:0/678452641 conn(0x7f4c3c0b35e0 msgr2=0x7f4c3c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:37.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.682+0000 7f4c4326a640 1 -- 172.21.15.37:0/678452641 shutdown_connections 2024-03-29T11:43:37.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.682+0000 7f4c4326a640 1 -- 172.21.15.37:0/678452641 wait complete. 2024-03-29T11:43:37.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.683+0000 7f4c4326a640 10 monclient: build_initial_monmap 2024-03-29T11:43:37.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.683+0000 7f4c4326a640 10 monclient: monmap: 2024-03-29T11:43:37.689 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:37.689 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:37.689 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:43:37.689 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:43:37.689 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:37.689 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:37.689 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:37.689 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:43:37.689 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:43:37.689 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:37.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.684+0000 7f4c4326a640 1 librados: starting msgr at 2024-03-29T11:43:37.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.684+0000 7f4c4326a640 1 librados: starting objecter 2024-03-29T11:43:37.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.684+0000 7f4c4326a640 1 Processor -- start 2024-03-29T11:43:37.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.684+0000 7f4c4326a640 1 -- start start 2024-03-29T11:43:37.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.684+0000 7f4c4326a640 1 librados: setting wanted keys 2024-03-29T11:43:37.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.684+0000 7f4c4326a640 1 librados: calling monclient init 2024-03-29T11:43:37.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.684+0000 7f4c4326a640 10 monclient: init 2024-03-29T11:43:37.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.685+0000 7f4c4326a640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:37.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.685+0000 7f4c4326a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:37.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.685+0000 7f4c4326a640 10 monclient(hunting): picked mon.noname-c con 0x7f4c3c0af900 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:37.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.685+0000 7f4c4326a640 10 monclient(hunting): picked mon.noname-a con 0x7f4c3c13ee90 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:37.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.685+0000 7f4c4326a640 10 monclient(hunting): picked mon.noname-b con 0x7f4c3c1c64f0 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:37.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.688+0000 7f4c4326a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3c06ec80 con 0x7f4c3c13ee90 2024-03-29T11:43:37.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.689+0000 7f4c4326a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3c070090 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.689+0000 7f4c4326a640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3c1c7c60 con 0x7f4c3c0af900 2024-03-29T11:43:37.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.689+0000 7f4c4326a640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:37.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.689+0000 7f4c4326a640 10 monclient(hunting): authenticate will time out at 1397.138156s 2024-03-29T11:43:37.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.689+0000 7f4c41807640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4c3c1c64f0 0x7f4c3c1ef920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:59437/0 (socket says 172.21.15.37:59437) 2024-03-29T11:43:37.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.689+0000 7f4c41807640 1 -- 172.21.15.37:0/1232670698 learned_addr learned my addr 172.21.15.37:0/1232670698 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:37.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.690+0000 7f4c31ffb640 10 client.?.objecter ms_handle_connect 0x7f4c3c1c64f0 2024-03-29T11:43:37.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.690+0000 7f4c31ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:37.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 10 client.?.objecter ms_handle_connect 0x7f4c3c0af900 2024-03-29T11:43:37.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206422085 0 0) 0x7f4c3c070090 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:43:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c20004eb0 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645090965 0 0) 0x7f4c3c1c7c60 con 0x7f4c3c0af900 2024-03-29T11:43:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:43:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c3c070090 con 0x7f4c3c0af900 2024-03-29T11:43:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 596639256 0 0) 0x7f4c20004eb0 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.692+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c3c1c7c60 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.693+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4c34001cd0 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.693+0000 7f4c31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:37.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.693+0000 7f4c31ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:37.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.693+0000 7f4c31ffb640 10 monclient(hunting): dump: 2024-03-29T11:43:37.700 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:37.700 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:37.700 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:37.700 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:37.700 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:37.700 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:37.700 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:37.700 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:37.700 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.693+0000 7f4c31ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.693+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3433052336 0 0) 0x7f4c3c070090 con 0x7f4c3c0af900 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.693+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c200069a0 con 0x7f4c3c0af900 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.693+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4c2c001510 con 0x7f4c3c0af900 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.693+0000 7f4c31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.693+0000 7f4c31ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.693+0000 7f4c31ffb640 10 monclient(hunting): dump: 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:37.701 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.693+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3403312937 0 0) 0x7f4c3c1c7c60 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 >> v1:172.21.15.37:6790/0 conn(0x7f4c3c0af900 legacy=0x7f4c3c1c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 >> v1:172.21.15.37:6789/0 conn(0x7f4c3c13ee90 legacy=0x7f4c3c1c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c31ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c31ffb640 1 monclient: found mon.b 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c31ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c06f110 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c31ffb640 10 monclient: _finish_auth 0 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c31ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c4326a640 5 monclient: authenticate success, global_id 4483 2024-03-29T11:43:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c4326a640 10 monclient: _renew_subs 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c4326a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c4326a640 1 -- 172.21.15.37:0/1232670698 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3c1c92f0 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c4326a640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c4326a640 10 monclient: _renew_subs 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c4326a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c4326a640 1 -- 172.21.15.37:0/1232670698 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3c06e980 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.694+0000 7f4c4326a640 1 librados: init done 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.695+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c34003970 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.695+0000 7f4c31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.695+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4c34001cd0 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.695+0000 7f4c31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.695+0000 7f4c31ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.695+0000 7f4c31ffb640 10 monclient: dump: 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:37.703 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:37.704 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:37.704 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:37.704 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:37.704 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:37.704 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:37.704 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:37.704 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:37.704 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:37.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.695+0000 7f4c337fe640 4 set_mon_vals no callback set 2024-03-29T11:43:37.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.697+0000 7f4c4326a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:37.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.697+0000 7f4c4326a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:37.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.697+0000 7f4c4326a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:37.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.697+0000 7f4c4326a640 1 -- 172.21.15.37:0/1232670698 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1c005080 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.697+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f4c34003be0 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.698+0000 7f4c41006640 10 monclient: get_auth_request con 0x7f4c2006e3d0 auth_method 0 2024-03-29T11:43:37.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.698+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f4c34001af0 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.698+0000 7f4c31ffb640 10 client.4483.objecter ms_dispatch 0x7f4c3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:37.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.698+0000 7f4c31ffb640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:37.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.698+0000 7f4c31ffb640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:37.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.698+0000 7f4c31ffb640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-29T11:43:37.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.699+0000 7f4c31ffb640 10 client.4483.objecter ms_handle_connect 0x7f4c2006e3d0 2024-03-29T11:43:37.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.705+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c34050a90 con 0x7f4c3c1c64f0 2024-03-29T11:43:37.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.705+0000 7f4c31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:37.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:37.705+0000 7f4c31ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:38.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.065+0000 7f4c4326a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:38.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.065+0000 7f4c4326a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:38.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.065+0000 7f4c4326a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:38.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.065+0000 7f4c4326a640 1 -- 172.21.15.37:0/1232670698 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c1c005330 con 0x7f4c3c1c64f0 2024-03-29T11:43:38.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.066+0000 7f4c31ffb640 1 -- 172.21.15.37:0/1232670698 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f4c34050c30 con 0x7f4c3c1c64f0 2024-03-29T11:43:38.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.066+0000 7f4c31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:38.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.066+0000 7f4c31ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:38.071 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:43:38.071 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:38.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.070+0000 7f4c4326a640 10 librados: watch_flush enter 2024-03-29T11:43:38.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.070+0000 7f4c4326a640 10 librados: watch_flush exit 2024-03-29T11:43:38.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.070+0000 7f4c4326a640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-29T11:43:38.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.070+0000 7f4c4326a640 10 client.4483.objecter successfully canceled tick 2024-03-29T11:43:38.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.070+0000 7f4c4326a640 1 -- 172.21.15.37:0/1232670698 >> v1:172.21.15.146:6816/2899149139 conn(0x7f4c2006e3d0 legacy=0x7f4c20070890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:38.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.070+0000 7f4c4326a640 10 monclient: shutdown 2024-03-29T11:43:38.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.071+0000 7f4c4326a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:38.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.071+0000 7f4c4326a640 1 -- 172.21.15.37:0/1232670698 >> v1:172.21.15.146:6789/0 conn(0x7f4c3c1c64f0 legacy=0x7f4c3c1ef920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:38.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.071+0000 7f4c4326a640 1 -- 172.21.15.37:0/1232670698 shutdown_connections 2024-03-29T11:43:38.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.071+0000 7f4c4326a640 1 -- 172.21.15.37:0/1232670698 >> 172.21.15.37:0/1232670698 conn(0x7f4c3c0b35e0 msgr2=0x7f4c3c139c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:38.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.071+0000 7f4c4326a640 1 -- 172.21.15.37:0/1232670698 shutdown_connections 2024-03-29T11:43:38.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.071+0000 7f4c4326a640 1 -- 172.21.15.37:0/1232670698 wait complete. 2024-03-29T11:43:38.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:38.071+0000 7f4c4326a640 1 librados: shutdown 2024-03-29T11:43:38.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:44.899 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:45.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.029+0000 7fcce905c640 10 monclient: get_monmap_and_config 2024-03-29T11:43:45.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.029+0000 7fcce905c640 10 monclient: build_initial_monmap 2024-03-29T11:43:45.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.029+0000 7fcce905c640 10 monclient: monmap: 2024-03-29T11:43:45.034 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:45.034 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:45.034 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:43:45.030784+0000 2024-03-29T11:43:45.034 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:43:45.030784+0000 2024-03-29T11:43:45.034 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:45.034 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:45.034 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:45.034 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:43:45.034 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:43:45.034 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:45.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.031+0000 7fcce905c640 1 Processor -- start 2024-03-29T11:43:45.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.031+0000 7fcce905c640 1 -- start start 2024-03-29T11:43:45.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.031+0000 7fcce905c640 10 monclient: init 2024-03-29T11:43:45.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.031+0000 7fcce905c640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:45.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.031+0000 7fcce905c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:45.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.031+0000 7fcce905c640 10 monclient(hunting): picked mon.noname-c con 0x7fcce40b0770 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.031+0000 7fcce905c640 10 monclient(hunting): picked mon.noname-b con 0x7fcce414a700 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.031+0000 7fcce905c640 10 monclient(hunting): picked mon.noname-a con 0x7fcce40ac5a0 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.031+0000 7fcce905c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce40aba70 con 0x7fcce40ac5a0 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce905c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce40aecb0 con 0x7fcce40b0770 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce905c640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce40af150 con 0x7fcce414a700 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce905c640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce905c640 10 monclient(hunting): authenticate will time out at 1404.481020s 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce259c640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fcce414a700 0x7fcce40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:41800/0 (socket says 172.21.15.37:41800) 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce259c640 1 -- 172.21.15.37:0/2195776033 learned_addr learned my addr 172.21.15.37:0/2195776033 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce1d9b640 1 -- 172.21.15.37:0/2195776033 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071177358 0 0) 0x7fcce40af150 con 0x7fcce414a700 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce1d9b640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:43:45.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce1d9b640 1 -- 172.21.15.37:0/2195776033 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcccc003520 con 0x7fcce414a700 2024-03-29T11:43:45.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce1d9b640 1 -- 172.21.15.37:0/2195776033 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1455410602 0 0) 0x7fcccc003520 con 0x7fcce414a700 2024-03-29T11:43:45.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce1d9b640 1 -- 172.21.15.37:0/2195776033 >> v1:172.21.15.146:6789/0 conn(0x7fcce40b0770 legacy=0x7fcce4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:45.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.032+0000 7fcce1d9b640 1 -- 172.21.15.37:0/2195776033 >> v1:172.21.15.37:6789/0 conn(0x7fcce40ac5a0 legacy=0x7fcce4151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:45.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.035+0000 7fcce1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:43:45.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.035+0000 7fcce1d9b640 1 monclient: found mon.noname-b 2024-03-29T11:43:45.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.035+0000 7fcce1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:45.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.035+0000 7fcce1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:43:45.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.035+0000 7fcce1d9b640 1 -- 172.21.15.37:0/2195776033 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce40ac9d0 con 0x7fcce414a700 2024-03-29T11:43:45.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.035+0000 7fcce1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:43:45.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.035+0000 7fcce1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.035+0000 7fcce1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.035+0000 7fcce905c640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.035+0000 7fcce1d9b640 1 -- 172.21.15.37:0/2195776033 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fccd4001690 con 0x7fcce414a700 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.035+0000 7fcce1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.035+0000 7fcce1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.035+0000 7fcce1d9b640 10 monclient: dump: 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:45.042 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.035+0000 7fcce905c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.036+0000 7fcce1d9b640 1 -- 172.21.15.37:0/2195776033 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccd40018f0 con 0x7fcce414a700 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.036+0000 7fcce1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.036+0000 7fcce1d9b640 1 -- 172.21.15.37:0/2195776033 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fccd4001d00 con 0x7fcce414a700 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.036+0000 7fcce1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.036+0000 7fcce1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.036+0000 7fcce1d9b640 10 monclient: dump: 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:45.043 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:45.044 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:45.044 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:45.044 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:45.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.036+0000 7fcce905c640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:45.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.036+0000 7fcce905c640 4 set_mon_vals no callback set 2024-03-29T11:43:45.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.037+0000 7fcce905c640 10 monclient: shutdown 2024-03-29T11:43:45.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.037+0000 7fcce905c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:45.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.037+0000 7fcce905c640 1 -- 172.21.15.37:0/2195776033 >> v1:172.21.15.37:6790/0 conn(0x7fcce414a700 legacy=0x7fcce40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:45.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.037+0000 7fcce905c640 1 -- 172.21.15.37:0/2195776033 shutdown_connections 2024-03-29T11:43:45.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.037+0000 7fcce905c640 1 -- 172.21.15.37:0/2195776033 >> 172.21.15.37:0/2195776033 conn(0x7fcce406b660 msgr2=0x7fcce406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:45.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.037+0000 7fcce905c640 1 -- 172.21.15.37:0/2195776033 shutdown_connections 2024-03-29T11:43:45.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.037+0000 7fcce905c640 1 -- 172.21.15.37:0/2195776033 wait complete. 2024-03-29T11:43:45.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.039+0000 7fcce905c640 10 monclient: build_initial_monmap 2024-03-29T11:43:45.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.039+0000 7fcce905c640 10 monclient: monmap: 2024-03-29T11:43:45.049 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:45.049 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:45.049 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:43:45.049 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:43:45.050 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:45.050 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:45.050 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:45.050 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:43:45.050 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:43:45.050 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:45.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.039+0000 7fcce905c640 1 librados: starting msgr at 2024-03-29T11:43:45.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.039+0000 7fcce905c640 1 librados: starting objecter 2024-03-29T11:43:45.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.040+0000 7fcce905c640 1 Processor -- start 2024-03-29T11:43:45.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.040+0000 7fcce905c640 1 -- start start 2024-03-29T11:43:45.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.040+0000 7fcce905c640 1 librados: setting wanted keys 2024-03-29T11:43:45.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.040+0000 7fcce905c640 1 librados: calling monclient init 2024-03-29T11:43:45.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.040+0000 7fcce905c640 10 monclient: init 2024-03-29T11:43:45.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.040+0000 7fcce905c640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.040+0000 7fcce905c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcce905c640 10 monclient(hunting): picked mon.noname-c con 0x7fcce40ac5a0 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcce905c640 10 monclient(hunting): picked mon.noname-b con 0x7fcce40b0770 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcce905c640 10 monclient(hunting): picked mon.noname-a con 0x7fcce414a700 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcce905c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce4128670 con 0x7fcce414a700 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcce905c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce4129fc0 con 0x7fcce40b0770 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcce905c640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce4147a50 con 0x7fcce40ac5a0 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcce905c640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcce905c640 10 monclient(hunting): authenticate will time out at 1404.490332s 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcce2d9d640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fcce40ac5a0 0x7fcce4128af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:56990/0 (socket says 172.21.15.37:56990) 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcce2d9d640 1 -- 172.21.15.37:0/702320985 learned_addr learned my addr 172.21.15.37:0/702320985 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcccb7fe640 10 client.?.objecter ms_handle_connect 0x7fcce40ac5a0 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcccb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135467976 0 0) 0x7fcce4147a50 con 0x7fcce40ac5a0 2024-03-29T11:43:45.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcccb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcccb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcccb7fe640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccd0004eb0 con 0x7fcce40ac5a0 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.041+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3895216869 0 0) 0x7fccd0004eb0 con 0x7fcce40ac5a0 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.042+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcce4147a50 con 0x7fcce40ac5a0 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.042+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fccd8004080 con 0x7fcce40ac5a0 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.042+0000 7fcccb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.042+0000 7fcccb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.042+0000 7fcccb7fe640 10 monclient(hunting): dump: 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:45.052 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.042+0000 7fcccb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.042+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 462539481 0 0) 0x7fcce4147a50 con 0x7fcce40ac5a0 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.042+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 >> v1:172.21.15.146:6789/0 conn(0x7fcce40b0770 legacy=0x7fcce41db740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.042+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 >> v1:172.21.15.37:6789/0 conn(0x7fcce414a700 legacy=0x7fcce41dbe50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.043+0000 7fcccb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.043+0000 7fcccb7fe640 1 monclient: found mon.c 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.043+0000 7fcccb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.043+0000 7fcccb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.043+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce4127bd0 con 0x7fcce40ac5a0 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.043+0000 7fcccb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:45.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.043+0000 7fcccb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.043+0000 7fcccb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.043+0000 7fcce905c640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.043+0000 7fcce905c640 10 monclient: _renew_subs 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.043+0000 7fcce905c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.043+0000 7fcce905c640 1 -- 172.21.15.37:0/702320985 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce41e6f70 con 0x7fcce40ac5a0 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.043+0000 7fcce905c640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.044+0000 7fcce905c640 10 monclient: _renew_subs 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.044+0000 7fcce905c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.044+0000 7fcce905c640 1 -- 172.21.15.37:0/702320985 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce41e72e0 con 0x7fcce40ac5a0 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.044+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccd80043a0 con 0x7fcce40ac5a0 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.044+0000 7fcccb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.044+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fccd8001320 con 0x7fcce40ac5a0 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.044+0000 7fcccb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.044+0000 7fcccb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:45.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.044+0000 7fcccb7fe640 10 monclient: dump: 2024-03-29T11:43:45.055 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:45.055 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:45.055 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:45.055 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:45.055 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:45.055 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:45.055 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:45.055 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:45.055 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:45.055 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:45.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.044+0000 7fcce159a640 4 set_mon_vals no callback set 2024-03-29T11:43:45.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.044+0000 7fcce905c640 1 librados: init done 2024-03-29T11:43:45.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.046+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fccd8001500 con 0x7fcce40ac5a0 2024-03-29T11:43:45.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.046+0000 7fccc97fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:45.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.047+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fccd8084b70 con 0x7fcce40ac5a0 2024-03-29T11:43:45.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.047+0000 7fcccb7fe640 10 client.4592.objecter ms_dispatch 0x7fcce4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:45.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.047+0000 7fcccb7fe640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:45.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.047+0000 7fcccb7fe640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:45.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.047+0000 7fcccb7fe640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-29T11:43:45.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.046+0000 7fccc97fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:45.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.047+0000 7fccc97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:45.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.047+0000 7fccc97fa640 1 -- 172.21.15.37:0/702320985 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccbc005080 con 0x7fcce40ac5a0 2024-03-29T11:43:45.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.047+0000 7fcce259c640 10 monclient: get_auth_request con 0x7fccd006dbc0 auth_method 0 2024-03-29T11:43:45.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.048+0000 7fcccb7fe640 10 client.4592.objecter ms_handle_connect 0x7fccd006dbc0 2024-03-29T11:43:45.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.054+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fccd8050450 con 0x7fcce40ac5a0 2024-03-29T11:43:45.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.054+0000 7fcccb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:45.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.054+0000 7fcccb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:45.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.368+0000 7fccc97fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:45.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.368+0000 7fccc97fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:45.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.368+0000 7fccc97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:45.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.368+0000 7fccc97fa640 1 -- 172.21.15.37:0/702320985 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccbc005330 con 0x7fcce40ac5a0 2024-03-29T11:43:45.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.369+0000 7fcccb7fe640 1 -- 172.21.15.37:0/702320985 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fccd8052380 con 0x7fcce40ac5a0 2024-03-29T11:43:45.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.369+0000 7fcccb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:45.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.369+0000 7fcccb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:45.373 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:43:45.373 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:45.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.372+0000 7fccc97fa640 10 librados: watch_flush enter 2024-03-29T11:43:45.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.373+0000 7fccc97fa640 10 librados: watch_flush exit 2024-03-29T11:43:45.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.373+0000 7fccc97fa640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-29T11:43:45.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.373+0000 7fccc97fa640 10 client.4592.objecter successfully canceled tick 2024-03-29T11:43:45.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.373+0000 7fccc97fa640 1 -- 172.21.15.37:0/702320985 >> v1:172.21.15.146:6816/2899149139 conn(0x7fccd006dbc0 legacy=0x7fccd0070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:45.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.373+0000 7fccc97fa640 10 monclient: shutdown 2024-03-29T11:43:45.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.373+0000 7fccc97fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:45.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.373+0000 7fccc97fa640 1 -- 172.21.15.37:0/702320985 >> v1:172.21.15.37:6790/0 conn(0x7fcce40ac5a0 legacy=0x7fcce4128af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:45.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.373+0000 7fccc97fa640 1 -- 172.21.15.37:0/702320985 shutdown_connections 2024-03-29T11:43:45.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.373+0000 7fccc97fa640 1 -- 172.21.15.37:0/702320985 >> 172.21.15.37:0/702320985 conn(0x7fcce406b660 msgr2=0x7fcce40abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:45.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.373+0000 7fccc97fa640 1 -- 172.21.15.37:0/702320985 shutdown_connections 2024-03-29T11:43:45.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.373+0000 7fccc97fa640 1 -- 172.21.15.37:0/702320985 wait complete. 2024-03-29T11:43:45.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:45.373+0000 7fccc97fa640 1 librados: shutdown 2024-03-29T11:43:45.392 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:52.200 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:52.350 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.347+0000 7fc862e64640 10 monclient: get_monmap_and_config 2024-03-29T11:43:52.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.347+0000 7fc862e64640 10 monclient: build_initial_monmap 2024-03-29T11:43:52.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.347+0000 7fc862e64640 10 monclient: monmap: 2024-03-29T11:43:52.351 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:52.351 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:52.351 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:43:52.348118+0000 2024-03-29T11:43:52.351 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:43:52.348118+0000 2024-03-29T11:43:52.351 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:52.351 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:52.351 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:52.351 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:43:52.351 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:43:52.352 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:52.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.348+0000 7fc862e64640 1 Processor -- start 2024-03-29T11:43:52.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.348+0000 7fc862e64640 1 -- start start 2024-03-29T11:43:52.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.348+0000 7fc862e64640 10 monclient: init 2024-03-29T11:43:52.355 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.349+0000 7fc862e64640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:52.355 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.349+0000 7fc862e64640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:52.355 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.349+0000 7fc862e64640 10 monclient(hunting): picked mon.noname-b con 0x7fc85c0b0770 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:52.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.349+0000 7fc862e64640 10 monclient(hunting): picked mon.noname-a con 0x7fc85c14a700 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:52.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.349+0000 7fc862e64640 10 monclient(hunting): picked mon.noname-c con 0x7fc85c0ac5a0 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:52.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.350+0000 7fc862e64640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85c0aba70 con 0x7fc85c14a700 2024-03-29T11:43:52.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.350+0000 7fc861401640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc85c0ac5a0 0x7fc85c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:59142/0 (socket says 172.21.15.37:59142) 2024-03-29T11:43:52.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.350+0000 7fc861401640 1 -- 172.21.15.37:0/2389519464 learned_addr learned my addr 172.21.15.37:0/2389519464 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:52.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.350+0000 7fc862e64640 1 -- 172.21.15.37:0/2389519464 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85c0aecb0 con 0x7fc85c0ac5a0 2024-03-29T11:43:52.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.350+0000 7fc85bfff640 1 --1- 172.21.15.37:0/2389519464 >> v1:172.21.15.37:6789/0 conn(0x7fc85c14a700 0x7fc85c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:58400/0 (socket says 172.21.15.37:58400) 2024-03-29T11:43:52.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.350+0000 7fc862e64640 1 -- 172.21.15.37:0/2389519464 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85c0af150 con 0x7fc85c0b0770 2024-03-29T11:43:52.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.350+0000 7fc862e64640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:52.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.350+0000 7fc862e64640 10 monclient(hunting): authenticate will time out at 1411.799735s 2024-03-29T11:43:52.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.350+0000 7fc85b7fe640 1 -- 172.21.15.37:0/2389519464 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520437915 0 0) 0x7fc85c0aecb0 con 0x7fc85c0ac5a0 2024-03-29T11:43:52.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.350+0000 7fc85b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.350+0000 7fc85b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.350+0000 7fc85b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.350+0000 7fc85b7fe640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.350+0000 7fc85b7fe640 1 -- 172.21.15.37:0/2389519464 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc844003520 con 0x7fc85c0ac5a0 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 1 -- 172.21.15.37:0/2389519464 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581397715 0 0) 0x7fc85c0aba70 con 0x7fc85c14a700 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 1 -- 172.21.15.37:0/2389519464 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc85c0aecb0 con 0x7fc85c14a700 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 1 -- 172.21.15.37:0/2389519464 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780078600 0 0) 0x7fc85c0af150 con 0x7fc85c0b0770 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 1 -- 172.21.15.37:0/2389519464 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc85c0aba70 con 0x7fc85c0b0770 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 1 -- 172.21.15.37:0/2389519464 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2518151761 0 0) 0x7fc85c0aecb0 con 0x7fc85c14a700 2024-03-29T11:43:52.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 1 -- 172.21.15.37:0/2389519464 >> v1:172.21.15.37:6790/0 conn(0x7fc85c0b0770 legacy=0x7fc85c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 1 -- 172.21.15.37:0/2389519464 >> v1:172.21.15.146:6789/0 conn(0x7fc85c0ac5a0 legacy=0x7fc85c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 1 -- 172.21.15.37:0/2389519464 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85c0ac9d0 con 0x7fc85c14a700 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 1 -- 172.21.15.37:0/2389519464 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc850001670 con 0x7fc85c14a700 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc85b7fe640 10 monclient: dump: 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:52.358 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:52.359 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:52.359 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:52.359 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.359 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:52.359 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:52.359 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:52.359 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:52.359 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:52.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc862e64640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:43:52.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.351+0000 7fc862e64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:52.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.352+0000 7fc85b7fe640 1 -- 172.21.15.37:0/2389519464 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8500017d0 con 0x7fc85c14a700 2024-03-29T11:43:52.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.353+0000 7fc85b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:52.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.353+0000 7fc85b7fe640 1 -- 172.21.15.37:0/2389519464 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc850001be0 con 0x7fc85c14a700 2024-03-29T11:43:52.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.353+0000 7fc85b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.353+0000 7fc85b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:52.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.353+0000 7fc85b7fe640 10 monclient: dump: 2024-03-29T11:43:52.367 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.353+0000 7fc862e64640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.353+0000 7fc862e64640 4 set_mon_vals no callback set 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.354+0000 7fc862e64640 10 monclient: shutdown 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.354+0000 7fc862e64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.354+0000 7fc862e64640 1 -- 172.21.15.37:0/2389519464 >> v1:172.21.15.37:6789/0 conn(0x7fc85c14a700 legacy=0x7fc85c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.354+0000 7fc862e64640 1 -- 172.21.15.37:0/2389519464 shutdown_connections 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.354+0000 7fc862e64640 1 -- 172.21.15.37:0/2389519464 >> 172.21.15.37:0/2389519464 conn(0x7fc85c06b660 msgr2=0x7fc85c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.354+0000 7fc862e64640 1 -- 172.21.15.37:0/2389519464 shutdown_connections 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.354+0000 7fc862e64640 1 -- 172.21.15.37:0/2389519464 wait complete. 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.355+0000 7fc862e64640 10 monclient: build_initial_monmap 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.355+0000 7fc862e64640 10 monclient: monmap: 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:43:52.369 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.355+0000 7fc862e64640 1 librados: starting msgr at 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.355+0000 7fc862e64640 1 librados: starting objecter 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.356+0000 7fc862e64640 1 Processor -- start 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.356+0000 7fc862e64640 1 -- start start 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.356+0000 7fc862e64640 1 librados: setting wanted keys 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.356+0000 7fc862e64640 1 librados: calling monclient init 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.356+0000 7fc862e64640 10 monclient: init 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.357+0000 7fc862e64640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.357+0000 7fc862e64640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.357+0000 7fc862e64640 10 monclient(hunting): picked mon.noname-c con 0x7fc85c0ac5a0 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.357+0000 7fc860c00640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fc85c0ac5a0 0x7fc85c128aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:59572/0 (socket says 172.21.15.37:59572) 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.358+0000 7fc860c00640 1 -- 172.21.15.37:0/1999242593 learned_addr learned my addr 172.21.15.37:0/1999242593 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.358+0000 7fc862e64640 10 monclient(hunting): picked mon.noname-b con 0x7fc85c0b0770 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.358+0000 7fc8597fa640 10 client.?.objecter ms_handle_connect 0x7fc85c0ac5a0 2024-03-29T11:43:52.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.358+0000 7fc8597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.358+0000 7fc8597fa640 10 client.?.objecter ms_handle_connect 0x7fc85c0b0770 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.358+0000 7fc8597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.358+0000 7fc862e64640 10 monclient(hunting): picked mon.noname-a con 0x7fc85c14a700 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.358+0000 7fc862e64640 1 -- 172.21.15.37:0/1999242593 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85c12a420 con 0x7fc85c14a700 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.358+0000 7fc862e64640 1 -- 172.21.15.37:0/1999242593 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85c1db9d0 con 0x7fc85c0b0770 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc862e64640 1 -- 172.21.15.37:0/1999242593 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85c1291b0 con 0x7fc85c0ac5a0 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc862e64640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc862e64640 10 monclient(hunting): authenticate will time out at 1411.808168s 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091368869 0 0) 0x7fc85c1db9d0 con 0x7fc85c0b0770 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc8597fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc8597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc8597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc8597fa640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc848004eb0 con 0x7fc85c0b0770 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc8597fa640 10 client.?.objecter ms_handle_connect 0x7fc85c14a700 2024-03-29T11:43:52.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc8597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:52.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076206757 0 0) 0x7fc85c1291b0 con 0x7fc85c0ac5a0 2024-03-29T11:43:52.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc8597fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc8597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc8597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc8597fa640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:43:52.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.359+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc85c1db9d0 con 0x7fc85c0ac5a0 2024-03-29T11:43:52.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.360+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215350071 0 0) 0x7fc85c12a420 con 0x7fc85c14a700 2024-03-29T11:43:52.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.360+0000 7fc8597fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.360+0000 7fc8597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.360+0000 7fc8597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.360+0000 7fc8597fa640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:43:52.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.360+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc85c1291b0 con 0x7fc85c14a700 2024-03-29T11:43:52.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.360+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1317133533 0 0) 0x7fc85c1db9d0 con 0x7fc85c0ac5a0 2024-03-29T11:43:52.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.360+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc85c12a420 con 0x7fc85c0ac5a0 2024-03-29T11:43:52.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.360+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc84c0019f0 con 0x7fc85c0ac5a0 2024-03-29T11:43:52.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.360+0000 7fc8597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.360+0000 7fc8597fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:52.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.360+0000 7fc8597fa640 10 monclient(hunting): dump: 2024-03-29T11:43:52.374 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:52.374 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:52.374 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:52.374 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:52.374 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.374 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.360+0000 7fc8597fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.360+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3321064516 0 0) 0x7fc85c1291b0 con 0x7fc85c14a700 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.361+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc85c1db9d0 con 0x7fc85c14a700 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.361+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc854001cb0 con 0x7fc85c14a700 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.361+0000 7fc8597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.361+0000 7fc8597fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.361+0000 7fc8597fa640 10 monclient(hunting): dump: 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:52.375 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.361+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3575973250 0 0) 0x7fc848004eb0 con 0x7fc85c0b0770 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.361+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc85c1291b0 con 0x7fc85c0b0770 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.361+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc850001b20 con 0x7fc85c0b0770 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.361+0000 7fc8597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.361+0000 7fc8597fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.361+0000 7fc8597fa640 10 monclient(hunting): dump: 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:52.376 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 863969471 0 0) 0x7fc85c1291b0 con 0x7fc85c0b0770 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 >> v1:172.21.15.37:6790/0 conn(0x7fc85c0ac5a0 legacy=0x7fc85c128aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 >> v1:172.21.15.37:6789/0 conn(0x7fc85c14a700 legacy=0x7fc85c1470c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc8597fa640 10 monclient: _finish_hunting 0 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc8597fa640 1 monclient: found mon.b 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc8597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc8597fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:52.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85c128030 con 0x7fc85c0b0770 2024-03-29T11:43:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc8597fa640 10 monclient: _finish_auth 0 2024-03-29T11:43:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc8597fa640 10 monclient: _check_auth_tickets 2024-03-29T11:43:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc8597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc862e64640 5 monclient: authenticate success, global_id 4489 2024-03-29T11:43:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc862e64640 10 monclient: _renew_subs 2024-03-29T11:43:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc862e64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc862e64640 1 -- 172.21.15.37:0/1999242593 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85c1dc600 con 0x7fc85c0b0770 2024-03-29T11:43:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc862e64640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.362+0000 7fc862e64640 10 monclient: _renew_subs 2024-03-29T11:43:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.363+0000 7fc862e64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.363+0000 7fc862e64640 1 -- 172.21.15.37:0/1999242593 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85c129600 con 0x7fc85c0b0770 2024-03-29T11:43:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.363+0000 7fc862e64640 1 librados: init done 2024-03-29T11:43:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.364+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc850001d20 con 0x7fc85c0b0770 2024-03-29T11:43:52.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.364+0000 7fc8597fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:52.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.364+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc850004c30 con 0x7fc85c0b0770 2024-03-29T11:43:52.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.364+0000 7fc8597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.364+0000 7fc8597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:52.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.364+0000 7fc8597fa640 10 monclient: dump: 2024-03-29T11:43:52.380 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:52.380 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:52.380 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:52.380 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.364+0000 7fc85affd640 4 set_mon_vals no callback set 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.364+0000 7fc862e64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.365+0000 7fc862e64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.365+0000 7fc862e64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.365+0000 7fc862e64640 1 -- 172.21.15.37:0/1999242593 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc83c005080 con 0x7fc85c0b0770 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.366+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fc85001ac00 con 0x7fc85c0b0770 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.367+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fc850084970 con 0x7fc85c0b0770 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.367+0000 7fc8597fa640 10 client.4489.objecter ms_dispatch 0x7fc85c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.367+0000 7fc8597fa640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:52.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.367+0000 7fc8597fa640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:52.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.367+0000 7fc8597fa640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-29T11:43:52.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.367+0000 7fc860c00640 10 monclient: get_auth_request con 0x7fc84806dff0 auth_method 0 2024-03-29T11:43:52.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.367+0000 7fc8597fa640 10 client.4489.objecter ms_handle_connect 0x7fc84806dff0 2024-03-29T11:43:52.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.375+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc850050390 con 0x7fc85c0b0770 2024-03-29T11:43:52.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.375+0000 7fc8597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:52.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.375+0000 7fc8597fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:52.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.761+0000 7fc862e64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:52.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.761+0000 7fc862e64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:52.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.761+0000 7fc862e64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:43:52.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.761+0000 7fc862e64640 1 -- 172.21.15.37:0/1999242593 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc83c005330 con 0x7fc85c0b0770 2024-03-29T11:43:52.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.762+0000 7fc8597fa640 1 -- 172.21.15.37:0/1999242593 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fc8500522c0 con 0x7fc85c0b0770 2024-03-29T11:43:52.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.763+0000 7fc8597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:52.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.763+0000 7fc8597fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:52.767 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:43:52.767 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:52.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.767+0000 7fc862e64640 10 librados: watch_flush enter 2024-03-29T11:43:52.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.767+0000 7fc862e64640 10 librados: watch_flush exit 2024-03-29T11:43:52.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.767+0000 7fc862e64640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-29T11:43:52.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.767+0000 7fc862e64640 10 client.4489.objecter successfully canceled tick 2024-03-29T11:43:52.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.767+0000 7fc862e64640 1 -- 172.21.15.37:0/1999242593 >> v1:172.21.15.146:6816/2899149139 conn(0x7fc84806dff0 legacy=0x7fc8480704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.767+0000 7fc862e64640 10 monclient: shutdown 2024-03-29T11:43:52.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.767+0000 7fc862e64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:52.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.767+0000 7fc862e64640 1 -- 172.21.15.37:0/1999242593 >> v1:172.21.15.146:6789/0 conn(0x7fc85c0b0770 legacy=0x7fc85c1dbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.767+0000 7fc862e64640 1 -- 172.21.15.37:0/1999242593 shutdown_connections 2024-03-29T11:43:52.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.767+0000 7fc862e64640 1 -- 172.21.15.37:0/1999242593 >> 172.21.15.37:0/1999242593 conn(0x7fc85c06b660 msgr2=0x7fc85c0abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:52.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.767+0000 7fc862e64640 1 -- 172.21.15.37:0/1999242593 shutdown_connections 2024-03-29T11:43:52.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.767+0000 7fc862e64640 1 -- 172.21.15.37:0/1999242593 wait complete. 2024-03-29T11:43:52.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:52.767+0000 7fc862e64640 1 librados: shutdown 2024-03-29T11:43:52.786 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:59.593 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:59.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.746+0000 7f8c3cc17640 10 monclient: get_monmap_and_config 2024-03-29T11:43:59.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.746+0000 7f8c3cc17640 10 monclient: build_initial_monmap 2024-03-29T11:43:59.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.746+0000 7f8c3cc17640 10 monclient: monmap: 2024-03-29T11:43:59.751 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:59.751 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:59.751 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:43:59.747541+0000 2024-03-29T11:43:59.751 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:43:59.747541+0000 2024-03-29T11:43:59.751 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:59.751 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:59.751 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:59.751 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:43:59.751 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:43:59.752 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:59.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.747+0000 7f8c3cc17640 1 Processor -- start 2024-03-29T11:43:59.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.747+0000 7f8c3cc17640 1 -- start start 2024-03-29T11:43:59.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.747+0000 7f8c3cc17640 10 monclient: init 2024-03-29T11:43:59.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.748+0000 7f8c3cc17640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:59.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.748+0000 7f8c3cc17640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:59.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.748+0000 7f8c3cc17640 10 monclient(hunting): picked mon.noname-b con 0x7f8c380a9ac0 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:59.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.748+0000 7f8c3cc17640 10 monclient(hunting): picked mon.noname-a con 0x7f8c3813ee90 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:59.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.748+0000 7f8c3cc17640 10 monclient(hunting): picked mon.noname-c con 0x7f8c380af900 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:59.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.748+0000 7f8c3cc17640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c380afd10 con 0x7f8c3813ee90 2024-03-29T11:43:59.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.748+0000 7f8c3cc17640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c380af6f0 con 0x7f8c380af900 2024-03-29T11:43:59.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.748+0000 7f8c3cc17640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c380aac80 con 0x7f8c380a9ac0 2024-03-29T11:43:59.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.748+0000 7f8c3cc17640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:59.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.748+0000 7f8c3cc17640 10 monclient(hunting): authenticate will time out at 1419.197914s 2024-03-29T11:43:59.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.753+0000 7f8c35d9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8c3813ee90 0x7f8c3813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:39491/0 (socket says 172.21.15.37:39491) 2024-03-29T11:43:59.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.753+0000 7f8c35d9b640 1 -- 172.21.15.37:0/2144671680 learned_addr learned my addr 172.21.15.37:0/2144671680 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:59.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.754+0000 7f8c3559a640 1 -- 172.21.15.37:0/2144671680 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552724020 0 0) 0x7f8c380afd10 con 0x7f8c3813ee90 2024-03-29T11:43:59.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.754+0000 7f8c3559a640 10 monclient(hunting): _negotiate 2024-03-29T11:43:59.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.754+0000 7f8c3559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:59.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.754+0000 7f8c3559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:59.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.754+0000 7f8c3559a640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:43:59.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.754+0000 7f8c3559a640 1 -- 172.21.15.37:0/2144671680 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c20003520 con 0x7f8c3813ee90 2024-03-29T11:43:59.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.754+0000 7f8c3559a640 1 -- 172.21.15.37:0/2144671680 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2707553481 0 0) 0x7f8c20003520 con 0x7f8c3813ee90 2024-03-29T11:43:59.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.755+0000 7f8c3559a640 1 -- 172.21.15.37:0/2144671680 >> v1:172.21.15.37:6790/0 conn(0x7f8c380a9ac0 legacy=0x7f8c380adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:59.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.755+0000 7f8c3559a640 1 -- 172.21.15.37:0/2144671680 >> v1:172.21.15.146:6789/0 conn(0x7f8c380af900 legacy=0x7f8c38149b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:43:59.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.755+0000 7f8c3559a640 10 monclient: _finish_hunting 0 2024-03-29T11:43:59.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.755+0000 7f8c3559a640 1 monclient: found mon.noname-a 2024-03-29T11:43:59.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.755+0000 7f8c3559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:59.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.755+0000 7f8c3559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:43:59.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.755+0000 7f8c3559a640 1 -- 172.21.15.37:0/2144671680 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c380aaf60 con 0x7f8c3813ee90 2024-03-29T11:43:59.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.755+0000 7f8c3559a640 10 monclient: _finish_auth 0 2024-03-29T11:43:59.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.755+0000 7f8c3559a640 10 monclient: _check_auth_tickets 2024-03-29T11:43:59.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.755+0000 7f8c3559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:59.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.755+0000 7f8c3cc17640 5 monclient: authenticate success, global_id 4536 2024-03-29T11:43:59.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.755+0000 7f8c3cc17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:59.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3559a640 1 -- 172.21.15.37:0/2144671680 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8c30001a10 con 0x7f8c3813ee90 2024-03-29T11:43:59.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:59.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:59.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3559a640 10 monclient: dump: 2024-03-29T11:43:59.761 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:59.761 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:59.761 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:59.761 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:59.761 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:59.762 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:59.762 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:59.762 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:59.762 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:59.762 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:59.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3cc17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:59.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3559a640 1 -- 172.21.15.37:0/2144671680 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c30001ca0 con 0x7f8c3813ee90 2024-03-29T11:43:59.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:59.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3cc17640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:59.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3cc17640 4 set_mon_vals no callback set 2024-03-29T11:43:59.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3559a640 1 -- 172.21.15.37:0/2144671680 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8c30002070 con 0x7f8c3813ee90 2024-03-29T11:43:59.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:59.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:59.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3559a640 10 monclient: dump: 2024-03-29T11:43:59.763 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:59.763 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:59.763 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:59.763 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:59.764 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:59.764 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:59.764 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:59.764 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:59.764 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:59.764 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:59.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3cc17640 10 monclient: shutdown 2024-03-29T11:43:59.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3cc17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:59.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.756+0000 7f8c3cc17640 1 -- 172.21.15.37:0/2144671680 >> v1:172.21.15.37:6789/0 conn(0x7f8c3813ee90 legacy=0x7f8c3813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:59.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.760+0000 7f8c3cc17640 1 -- 172.21.15.37:0/2144671680 shutdown_connections 2024-03-29T11:43:59.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.760+0000 7f8c3cc17640 1 -- 172.21.15.37:0/2144671680 >> 172.21.15.37:0/2144671680 conn(0x7f8c380b35e0 msgr2=0x7f8c380b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:59.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.760+0000 7f8c3cc17640 1 -- 172.21.15.37:0/2144671680 shutdown_connections 2024-03-29T11:43:59.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.760+0000 7f8c3cc17640 1 -- 172.21.15.37:0/2144671680 wait complete. 2024-03-29T11:43:59.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.761+0000 7f8c3cc17640 10 monclient: build_initial_monmap 2024-03-29T11:43:59.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.761+0000 7f8c3cc17640 10 monclient: monmap: 2024-03-29T11:43:59.766 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:43:59.766 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:59.766 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:43:59.766 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:43:59.766 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:59.766 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:59.766 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:43:59.766 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:43:59.766 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:43:59.766 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:59.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.761+0000 7f8c3cc17640 1 librados: starting msgr at 2024-03-29T11:43:59.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.761+0000 7f8c3cc17640 1 librados: starting objecter 2024-03-29T11:43:59.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.761+0000 7f8c3cc17640 1 Processor -- start 2024-03-29T11:43:59.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.762+0000 7f8c3cc17640 1 -- start start 2024-03-29T11:43:59.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.762+0000 7f8c3cc17640 1 librados: setting wanted keys 2024-03-29T11:43:59.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.762+0000 7f8c3cc17640 1 librados: calling monclient init 2024-03-29T11:43:59.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.762+0000 7f8c3cc17640 10 monclient: init 2024-03-29T11:43:59.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.762+0000 7f8c3cc17640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:59.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.762+0000 7f8c3cc17640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:59.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.762+0000 7f8c3cc17640 10 monclient(hunting): picked mon.noname-a con 0x7f8c380a9ac0 addr v1:172.21.15.37:6789/0 2024-03-29T11:43:59.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.762+0000 7f8c3cc17640 10 monclient(hunting): picked mon.noname-b con 0x7f8c380af900 addr v1:172.21.15.146:6789/0 2024-03-29T11:43:59.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.762+0000 7f8c3cc17640 10 monclient(hunting): picked mon.noname-c con 0x7f8c3813ee90 addr v1:172.21.15.37:6790/0 2024-03-29T11:43:59.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.762+0000 7f8c3cc17640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c3813faa0 con 0x7f8c380a9ac0 2024-03-29T11:43:59.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.762+0000 7f8c3cc17640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c38140f70 con 0x7f8c380af900 2024-03-29T11:43:59.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.762+0000 7f8c3cc17640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c3813f5c0 con 0x7f8c3813ee90 2024-03-29T11:43:59.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.762+0000 7f8c3cc17640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:59.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.762+0000 7f8c3cc17640 10 monclient(hunting): authenticate will time out at 1419.211847s 2024-03-29T11:43:59.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.767+0000 7f8c36d9d640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f8c3813ee90 0x7f8c381e6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:59881/0 (socket says 172.21.15.37:59881) 2024-03-29T11:43:59.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.767+0000 7f8c36d9d640 1 -- 172.21.15.37:0/982054273 learned_addr learned my addr 172.21.15.37:0/982054273 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:43:59.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.767+0000 7f8c26ffd640 10 client.?.objecter ms_handle_connect 0x7f8c3813ee90 2024-03-29T11:43:59.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.767+0000 7f8c26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:59.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.768+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806603100 0 0) 0x7f8c3813f5c0 con 0x7f8c3813ee90 2024-03-29T11:43:59.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.768+0000 7f8c26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:59.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.768+0000 7f8c26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:59.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.768+0000 7f8c26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:59.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.768+0000 7f8c26ffd640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:43:59.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.768+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c18004eb0 con 0x7f8c3813ee90 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.769+0000 7f8c26ffd640 10 client.?.objecter ms_handle_connect 0x7f8c380af900 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.769+0000 7f8c26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.769+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2759138516 0 0) 0x7f8c18004eb0 con 0x7f8c3813ee90 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.769+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c18005300 con 0x7f8c3813ee90 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.769+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8c280017b0 con 0x7f8c3813ee90 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.769+0000 7f8c26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.770+0000 7f8c26ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.770+0000 7f8c26ffd640 10 monclient(hunting): dump: 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:59.776 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.770+0000 7f8c26ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.770+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323627629 0 0) 0x7f8c38140f70 con 0x7f8c380af900 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.770+0000 7f8c26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.770+0000 7f8c26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.770+0000 7f8c26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.770+0000 7f8c26ffd640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.770+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c18004eb0 con 0x7f8c380af900 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.770+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4189435531 0 0) 0x7f8c18005300 con 0x7f8c3813ee90 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 >> v1:172.21.15.146:6789/0 conn(0x7f8c380af900 legacy=0x7f8c381de9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 >> v1:172.21.15.37:6789/0 conn(0x7f8c380a9ac0 legacy=0x7f8c3812ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c26ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:59.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c26ffd640 1 monclient: found mon.c 2024-03-29T11:43:59.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:59.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c26ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:59.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3813f750 con 0x7f8c3813ee90 2024-03-29T11:43:59.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c26ffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:59.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c26ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:59.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:59.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c3cc17640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:43:59.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c3cc17640 10 monclient: _renew_subs 2024-03-29T11:43:59.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c3cc17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:59.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c3cc17640 1 -- 172.21.15.37:0/982054273 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c3814b490 con 0x7f8c3813ee90 2024-03-29T11:43:59.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c3cc17640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:59.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c3cc17640 10 monclient: _renew_subs 2024-03-29T11:43:59.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c3cc17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:59.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c3cc17640 1 -- 172.21.15.37:0/982054273 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c3813d980 con 0x7f8c3813ee90 2024-03-29T11:43:59.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.771+0000 7f8c3cc17640 1 librados: init done 2024-03-29T11:43:59.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.773+0000 7f8c3cc17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:59.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.773+0000 7f8c3cc17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:59.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.773+0000 7f8c3cc17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:43:59.780 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.773+0000 7f8c3cc17640 1 -- 172.21.15.37:0/982054273 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c3813bd00 con 0x7f8c3813ee90 2024-03-29T11:43:59.781 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.774+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c28001d40 con 0x7f8c3813ee90 2024-03-29T11:43:59.781 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.774+0000 7f8c26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:59.781 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.774+0000 7f8c34d99640 4 set_mon_vals no callback set 2024-03-29T11:43:59.781 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.774+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8c28004900 con 0x7f8c3813ee90 2024-03-29T11:43:59.781 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.774+0000 7f8c26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:59.781 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.774+0000 7f8c26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:59.781 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.774+0000 7f8c26ffd640 10 monclient: dump: 2024-03-29T11:43:59.781 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:43:59.781 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:43:59.781 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:59.782 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:43:59.782 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:59.782 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:43:59.782 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:43:59.782 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:43:59.782 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:43:59.782 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:43:59.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.778+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f8c2801a8d0 con 0x7f8c3813ee90 2024-03-29T11:43:59.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.779+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f8c28054340 con 0x7f8c3813ee90 2024-03-29T11:43:59.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.779+0000 7f8c26ffd640 10 client.4601.objecter ms_dispatch 0x7f8c38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:59.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.779+0000 7f8c26ffd640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:59.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.779+0000 7f8c26ffd640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:59.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.779+0000 7f8c26ffd640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-29T11:43:59.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.779+0000 7f8c3659c640 10 monclient: get_auth_request con 0x7f8c1806e1d0 auth_method 0 2024-03-29T11:43:59.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.780+0000 7f8c26ffd640 10 client.4601.objecter ms_handle_connect 0x7f8c1806e1d0 2024-03-29T11:43:59.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.785+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c280505e0 con 0x7f8c3813ee90 2024-03-29T11:43:59.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.785+0000 7f8c26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:59.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:43:59.785+0000 7f8c26ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:00.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.119+0000 7f8c3cc17640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:00.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.119+0000 7f8c3cc17640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:00.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.119+0000 7f8c3cc17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:44:00.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.119+0000 7f8c3cc17640 1 -- 172.21.15.37:0/982054273 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c3813bd00 con 0x7f8c3813ee90 2024-03-29T11:44:00.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.120+0000 7f8c26ffd640 1 -- 172.21.15.37:0/982054273 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f8c280505e0 con 0x7f8c3813ee90 2024-03-29T11:44:00.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.120+0000 7f8c26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:00.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.120+0000 7f8c26ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:00.126 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:44:00.127 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:00.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.125+0000 7f8c3cc17640 10 librados: watch_flush enter 2024-03-29T11:44:00.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.125+0000 7f8c3cc17640 10 librados: watch_flush exit 2024-03-29T11:44:00.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.125+0000 7f8c3cc17640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-29T11:44:00.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.125+0000 7f8c3cc17640 10 client.4601.objecter successfully canceled tick 2024-03-29T11:44:00.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.125+0000 7f8c3cc17640 1 -- 172.21.15.37:0/982054273 >> v1:172.21.15.146:6816/2899149139 conn(0x7f8c1806e1d0 legacy=0x7f8c18070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:00.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.125+0000 7f8c3cc17640 10 monclient: shutdown 2024-03-29T11:44:00.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.125+0000 7f8c3cc17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:00.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.125+0000 7f8c3cc17640 1 -- 172.21.15.37:0/982054273 >> v1:172.21.15.37:6790/0 conn(0x7f8c3813ee90 legacy=0x7f8c381e6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:00.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.125+0000 7f8c3cc17640 1 -- 172.21.15.37:0/982054273 shutdown_connections 2024-03-29T11:44:00.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.125+0000 7f8c3cc17640 1 -- 172.21.15.37:0/982054273 >> 172.21.15.37:0/982054273 conn(0x7f8c380b35e0 msgr2=0x7f8c380acf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:00.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.125+0000 7f8c3cc17640 1 -- 172.21.15.37:0/982054273 shutdown_connections 2024-03-29T11:44:00.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.125+0000 7f8c3cc17640 1 -- 172.21.15.37:0/982054273 wait complete. 2024-03-29T11:44:00.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:00.126+0000 7f8c3cc17640 1 librados: shutdown 2024-03-29T11:44:00.146 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:06.954 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:07.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.107+0000 7f01849bc640 10 monclient: get_monmap_and_config 2024-03-29T11:44:07.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.107+0000 7f01849bc640 10 monclient: build_initial_monmap 2024-03-29T11:44:07.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.107+0000 7f01849bc640 10 monclient: monmap: 2024-03-29T11:44:07.113 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:07.113 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:07.113 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:44:07.108233+0000 2024-03-29T11:44:07.113 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:44:07.108233+0000 2024-03-29T11:44:07.113 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:07.113 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:07.113 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:07.114 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:44:07.114 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:44:07.114 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:07.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.108+0000 7f01849bc640 1 Processor -- start 2024-03-29T11:44:07.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.108+0000 7f01849bc640 1 -- start start 2024-03-29T11:44:07.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.108+0000 7f01849bc640 10 monclient: init 2024-03-29T11:44:07.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.109+0000 7f01849bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:07.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.109+0000 7f01849bc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:07.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.111+0000 7f01849bc640 10 monclient(hunting): picked mon.noname-b con 0x7f0178029eb0 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:07.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.111+0000 7f01849bc640 10 monclient(hunting): picked mon.noname-c con 0x7f017802b0a0 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:07.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.111+0000 7f01849bc640 10 monclient(hunting): picked mon.noname-a con 0x7f01780c9570 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:07.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.111+0000 7f01849bc640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017802a290 con 0x7f01780c9570 2024-03-29T11:44:07.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.111+0000 7f01849bc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01780331c0 con 0x7f017802b0a0 2024-03-29T11:44:07.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.111+0000 7f01849bc640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017802c100 con 0x7f0178029eb0 2024-03-29T11:44:07.116 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.111+0000 7f01849bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:07.116 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.111+0000 7f01849bc640 10 monclient(hunting): authenticate will time out at 1426.560708s 2024-03-29T11:44:07.117 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.113+0000 7f017f7fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f0178029eb0 0x7f01780cf120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:52711/0 (socket says 172.21.15.37:52711) 2024-03-29T11:44:07.117 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.113+0000 7f017f7fe640 1 -- 172.21.15.37:0/543466020 learned_addr learned my addr 172.21.15.37:0/543466020 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:07.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.115+0000 7f017e7fc640 1 -- 172.21.15.37:0/543466020 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095748798 0 0) 0x7f017802c100 con 0x7f0178029eb0 2024-03-29T11:44:07.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.115+0000 7f017e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:07.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.115+0000 7f017e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:07.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.115+0000 7f017e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:07.119 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.115+0000 7f017e7fc640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:44:07.119 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.115+0000 7f017e7fc640 1 -- 172.21.15.37:0/543466020 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0170003520 con 0x7f0178029eb0 2024-03-29T11:44:07.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.116+0000 7f017e7fc640 1 -- 172.21.15.37:0/543466020 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2367501498 0 0) 0x7f0170003520 con 0x7f0178029eb0 2024-03-29T11:44:07.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.116+0000 7f017e7fc640 1 -- 172.21.15.37:0/543466020 >> v1:172.21.15.146:6789/0 conn(0x7f017802b0a0 legacy=0x7f01780c8e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:07.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.116+0000 7f017e7fc640 1 -- 172.21.15.37:0/543466020 >> v1:172.21.15.37:6789/0 conn(0x7f01780c9570 legacy=0x7f01780d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:07.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.116+0000 7f017e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:07.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.116+0000 7f017e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:44:07.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.116+0000 7f017e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:07.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.116+0000 7f017e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:44:07.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.116+0000 7f017e7fc640 1 -- 172.21.15.37:0/543466020 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017802a440 con 0x7f0178029eb0 2024-03-29T11:44:07.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.117+0000 7f017e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:07.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.117+0000 7f017e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:07.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.117+0000 7f017e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:07.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.117+0000 7f017e7fc640 1 -- 172.21.15.37:0/543466020 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f016c001690 con 0x7f0178029eb0 2024-03-29T11:44:07.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.117+0000 7f017e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:07.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.117+0000 7f017e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:07.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.117+0000 7f017e7fc640 10 monclient: dump: 2024-03-29T11:44:07.122 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:07.122 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:07.122 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:07.122 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:07.122 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:07.122 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:07.122 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:07.122 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:07.122 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:07.122 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:07.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.117+0000 7f01849bc640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:44:07.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.117+0000 7f01849bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:07.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.120+0000 7f017e7fc640 1 -- 172.21.15.37:0/543466020 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f016c001860 con 0x7f0178029eb0 2024-03-29T11:44:07.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.120+0000 7f017e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:07.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.120+0000 7f01849bc640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:07.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.120+0000 7f01849bc640 4 set_mon_vals no callback set 2024-03-29T11:44:07.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.121+0000 7f01849bc640 10 monclient: shutdown 2024-03-29T11:44:07.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.121+0000 7f01849bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:07.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.121+0000 7f01849bc640 1 -- 172.21.15.37:0/543466020 >> v1:172.21.15.37:6790/0 conn(0x7f0178029eb0 legacy=0x7f01780cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:07.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.122+0000 7f01849bc640 1 -- 172.21.15.37:0/543466020 shutdown_connections 2024-03-29T11:44:07.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.122+0000 7f01849bc640 1 -- 172.21.15.37:0/543466020 >> 172.21.15.37:0/543466020 conn(0x7f01780058f0 msgr2=0x7f0178005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:07.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.122+0000 7f01849bc640 1 -- 172.21.15.37:0/543466020 shutdown_connections 2024-03-29T11:44:07.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.125+0000 7f01849bc640 1 -- 172.21.15.37:0/543466020 wait complete. 2024-03-29T11:44:07.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.126+0000 7f01849bc640 10 monclient: build_initial_monmap 2024-03-29T11:44:07.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.126+0000 7f01849bc640 10 monclient: monmap: 2024-03-29T11:44:07.130 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:07.130 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:07.130 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:44:07.130 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:44:07.130 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:07.130 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:07.130 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:07.130 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:44:07.130 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:44:07.130 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:07.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.126+0000 7f01849bc640 1 librados: starting msgr at 2024-03-29T11:44:07.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.126+0000 7f01849bc640 1 librados: starting objecter 2024-03-29T11:44:07.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.127+0000 7f01849bc640 1 Processor -- start 2024-03-29T11:44:07.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.127+0000 7f01849bc640 1 -- start start 2024-03-29T11:44:07.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.127+0000 7f01849bc640 1 librados: setting wanted keys 2024-03-29T11:44:07.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.127+0000 7f01849bc640 1 librados: calling monclient init 2024-03-29T11:44:07.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.127+0000 7f01849bc640 10 monclient: init 2024-03-29T11:44:07.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.128+0000 7f01849bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:07.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.128+0000 7f01849bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:07.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.128+0000 7f01849bc640 10 monclient(hunting): picked mon.noname-b con 0x7f0178029eb0 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:07.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.128+0000 7f01849bc640 10 monclient(hunting): picked mon.noname-c con 0x7f017802b0a0 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:07.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.131+0000 7f01849bc640 10 monclient(hunting): picked mon.noname-a con 0x7f01780c9570 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:07.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.131+0000 7f01849bc640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01780311d0 con 0x7f01780c9570 2024-03-29T11:44:07.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.131+0000 7f01849bc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0178031400 con 0x7f0178029eb0 2024-03-29T11:44:07.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.131+0000 7f01849bc640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0178160bb0 con 0x7f017802b0a0 2024-03-29T11:44:07.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.132+0000 7f01849bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:07.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.132+0000 7f01849bc640 10 monclient(hunting): authenticate will time out at 1426.581057s 2024-03-29T11:44:07.138 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.134+0000 7f017f7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0178029eb0 0x7f017802eda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:50509/0 (socket says 172.21.15.37:50509) 2024-03-29T11:44:07.138 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.134+0000 7f017f7fe640 1 -- 172.21.15.37:0/1359232018 learned_addr learned my addr 172.21.15.37:0/1359232018 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:07.138 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.135+0000 7f016bfff640 10 client.?.objecter ms_handle_connect 0x7f01780c9570 2024-03-29T11:44:07.138 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.135+0000 7f016bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:07.139 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.135+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869732155 0 0) 0x7f01780311d0 con 0x7f01780c9570 2024-03-29T11:44:07.139 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.135+0000 7f016bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:07.139 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.135+0000 7f016bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:07.139 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.135+0000 7f016bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:07.139 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.135+0000 7f016bfff640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:44:07.139 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.135+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0164004eb0 con 0x7f01780c9570 2024-03-29T11:44:07.140 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.135+0000 7f016bfff640 10 client.?.objecter ms_handle_connect 0x7f017802b0a0 2024-03-29T11:44:07.140 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.135+0000 7f016bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:07.140 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4243716095 0 0) 0x7f0164004eb0 con 0x7f01780c9570 2024-03-29T11:44:07.140 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01780311d0 con 0x7f01780c9570 2024-03-29T11:44:07.140 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f018004fdd0 con 0x7f01780c9570 2024-03-29T11:44:07.140 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:07.140 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:07.140 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 10 monclient(hunting): dump: 2024-03-29T11:44:07.140 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:07.140 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:07.140 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:07.140 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:07.141 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:07.141 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:07.141 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:07.141 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:07.141 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:07.141 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:07.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:07.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779104571 0 0) 0x7f0178160bb0 con 0x7f017802b0a0 2024-03-29T11:44:07.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:07.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:07.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:07.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:44:07.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0164004eb0 con 0x7f017802b0a0 2024-03-29T11:44:07.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1919657772 0 0) 0x7f01780311d0 con 0x7f01780c9570 2024-03-29T11:44:07.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 >> v1:172.21.15.37:6790/0 conn(0x7f017802b0a0 legacy=0x7f01781610d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:07.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 >> v1:172.21.15.146:6789/0 conn(0x7f0178029eb0 legacy=0x7f017802eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:07.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:07.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 1 monclient: found mon.a 2024-03-29T11:44:07.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:07.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:07.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01781617e0 con 0x7f01780c9570 2024-03-29T11:44:07.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 10 monclient: _finish_auth 0 2024-03-29T11:44:07.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:07.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:07.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 10 client.?.objecter ms_handle_connect 0x7f0178029eb0 2024-03-29T11:44:07.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f016bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:07.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f01849bc640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:44:07.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f01849bc640 10 monclient: _renew_subs 2024-03-29T11:44:07.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f01849bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:07.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.136+0000 7f01849bc640 1 -- 172.21.15.37:0/1359232018 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01780c4db0 con 0x7f01780c9570 2024-03-29T11:44:07.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.137+0000 7f01849bc640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:07.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.137+0000 7f01849bc640 10 monclient: _renew_subs 2024-03-29T11:44:07.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.137+0000 7f01849bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:07.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.137+0000 7f01849bc640 1 -- 172.21.15.37:0/1359232018 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0178031a00 con 0x7f01780c9570 2024-03-29T11:44:07.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.137+0000 7f01849bc640 1 librados: init done 2024-03-29T11:44:07.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.138+0000 7f01849bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:07.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.139+0000 7f01849bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:07.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.139+0000 7f01849bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:07.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.139+0000 7f01849bc640 1 -- 172.21.15.37:0/1359232018 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017802ad00 con 0x7f01780c9570 2024-03-29T11:44:07.147 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.142+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f018004f400 con 0x7f01780c9570 2024-03-29T11:44:07.147 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.142+0000 7f016bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:07.147 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.142+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0180075f10 con 0x7f01780c9570 2024-03-29T11:44:07.147 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.142+0000 7f016bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:07.147 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.142+0000 7f016bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.142+0000 7f016bfff640 10 monclient: dump: 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.142+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f0180067bf0 con 0x7f01780c9570 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.142+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f01800fe310 con 0x7f01780c9570 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.142+0000 7f016bfff640 10 client.4542.objecter ms_dispatch 0x7f0180000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:07.148 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.142+0000 7f016bfff640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:07.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.142+0000 7f016bfff640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:07.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.142+0000 7f016bfff640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-29T11:44:07.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.142+0000 7f017d7fa640 4 set_mon_vals no callback set 2024-03-29T11:44:07.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.143+0000 7f017f7fe640 10 monclient: get_auth_request con 0x7f016406dc80 auth_method 0 2024-03-29T11:44:07.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.144+0000 7f016bfff640 10 client.4542.objecter ms_handle_connect 0x7f016406dc80 2024-03-29T11:44:07.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.150+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01800c9cd0 con 0x7f01780c9570 2024-03-29T11:44:07.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.150+0000 7f016bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:07.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.150+0000 7f016bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:07.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.480+0000 7f01849bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:07.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.480+0000 7f01849bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:07.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.480+0000 7f01849bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:07.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.480+0000 7f01849bc640 1 -- 172.21.15.37:0/1359232018 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0178160a10 con 0x7f01780c9570 2024-03-29T11:44:07.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.481+0000 7f016bfff640 1 -- 172.21.15.37:0/1359232018 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f0178160a10 con 0x7f01780c9570 2024-03-29T11:44:07.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.481+0000 7f016bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:07.485 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.481+0000 7f016bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:07.485 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:44:07.485 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:07.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.484+0000 7f01849bc640 10 librados: watch_flush enter 2024-03-29T11:44:07.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.485+0000 7f01849bc640 10 librados: watch_flush exit 2024-03-29T11:44:07.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.485+0000 7f01849bc640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-29T11:44:07.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.485+0000 7f01849bc640 10 client.4542.objecter successfully canceled tick 2024-03-29T11:44:07.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.485+0000 7f01849bc640 1 -- 172.21.15.37:0/1359232018 >> v1:172.21.15.146:6816/2899149139 conn(0x7f016406dc80 legacy=0x7f0164070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:07.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.485+0000 7f01849bc640 10 monclient: shutdown 2024-03-29T11:44:07.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.485+0000 7f01849bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:07.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.485+0000 7f01849bc640 1 -- 172.21.15.37:0/1359232018 >> v1:172.21.15.37:6789/0 conn(0x7f01780c9570 legacy=0x7f01780c46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:07.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.485+0000 7f01849bc640 1 -- 172.21.15.37:0/1359232018 shutdown_connections 2024-03-29T11:44:07.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.485+0000 7f01849bc640 1 -- 172.21.15.37:0/1359232018 >> 172.21.15.37:0/1359232018 conn(0x7f01780058f0 msgr2=0x7f01780043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:07.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.485+0000 7f01849bc640 1 -- 172.21.15.37:0/1359232018 shutdown_connections 2024-03-29T11:44:07.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.486+0000 7f01849bc640 1 -- 172.21.15.37:0/1359232018 wait complete. 2024-03-29T11:44:07.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:07.487+0000 7f01849bc640 1 librados: shutdown 2024-03-29T11:44:07.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:14.316 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:14.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.464+0000 7fbf9be2e640 10 monclient: get_monmap_and_config 2024-03-29T11:44:14.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.464+0000 7fbf9be2e640 10 monclient: build_initial_monmap 2024-03-29T11:44:14.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.464+0000 7fbf9be2e640 10 monclient: monmap: 2024-03-29T11:44:14.469 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:14.469 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:14.470 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:44:14.465527+0000 2024-03-29T11:44:14.470 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:44:14.465527+0000 2024-03-29T11:44:14.470 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:14.470 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:14.470 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:14.470 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:44:14.470 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:44:14.470 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:14.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.466+0000 7fbf9be2e640 1 Processor -- start 2024-03-29T11:44:14.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.467+0000 7fbf9be2e640 1 -- start start 2024-03-29T11:44:14.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.467+0000 7fbf9be2e640 10 monclient: init 2024-03-29T11:44:14.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.468+0000 7fbf9be2e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:14.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.468+0000 7fbf9be2e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:14.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.468+0000 7fbf9be2e640 10 monclient(hunting): picked mon.noname-b con 0x7fbf940a9ac0 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:14.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.468+0000 7fbf9be2e640 10 monclient(hunting): picked mon.noname-c con 0x7fbf9413ee90 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:14.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.468+0000 7fbf9be2e640 10 monclient(hunting): picked mon.noname-a con 0x7fbf940af900 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:14.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.468+0000 7fbf9be2e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf940afd10 con 0x7fbf940af900 2024-03-29T11:44:14.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.468+0000 7fbf9be2e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf940af6f0 con 0x7fbf9413ee90 2024-03-29T11:44:14.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.468+0000 7fbf9be2e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf940aac80 con 0x7fbf940a9ac0 2024-03-29T11:44:14.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.468+0000 7fbf9be2e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:14.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.468+0000 7fbf9be2e640 10 monclient(hunting): authenticate will time out at 1433.917191s 2024-03-29T11:44:14.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.468+0000 7fbf99bca640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fbf940a9ac0 0x7fbf940adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:49194/0 (socket says 172.21.15.37:49194) 2024-03-29T11:44:14.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.468+0000 7fbf99bca640 1 -- 172.21.15.37:0/1232031157 learned_addr learned my addr 172.21.15.37:0/1232031157 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:14.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.469+0000 7fbf98bc8640 1 -- 172.21.15.37:0/1232031157 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584333494 0 0) 0x7fbf940aac80 con 0x7fbf940a9ac0 2024-03-29T11:44:14.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.469+0000 7fbf98bc8640 10 monclient(hunting): _negotiate 2024-03-29T11:44:14.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.469+0000 7fbf98bc8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:14.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.469+0000 7fbf98bc8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:14.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.469+0000 7fbf98bc8640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:44:14.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.469+0000 7fbf98bc8640 1 -- 172.21.15.37:0/1232031157 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf80003520 con 0x7fbf940a9ac0 2024-03-29T11:44:14.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.471+0000 7fbf98bc8640 1 -- 172.21.15.37:0/1232031157 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 78414721 0 0) 0x7fbf80003520 con 0x7fbf940a9ac0 2024-03-29T11:44:14.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.471+0000 7fbf98bc8640 1 -- 172.21.15.37:0/1232031157 >> v1:172.21.15.146:6789/0 conn(0x7fbf9413ee90 legacy=0x7fbf9413b5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:44:14.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.471+0000 7fbf98bc8640 1 -- 172.21.15.37:0/1232031157 >> v1:172.21.15.37:6789/0 conn(0x7fbf940af900 legacy=0x7fbf94149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:14.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 10 monclient: _finish_hunting 0 2024-03-29T11:44:14.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 1 monclient: found mon.noname-b 2024-03-29T11:44:14.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:14.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:44:14.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 1 -- 172.21.15.37:0/1232031157 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf940aaf60 con 0x7fbf940a9ac0 2024-03-29T11:44:14.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 10 monclient: _finish_auth 0 2024-03-29T11:44:14.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 10 monclient: _check_auth_tickets 2024-03-29T11:44:14.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:14.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 1 -- 172.21.15.37:0/1232031157 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbf90001690 con 0x7fbf940a9ac0 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 10 monclient: dump: 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf9be2e640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf9be2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:14.479 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 1 -- 172.21.15.37:0/1232031157 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf90001860 con 0x7fbf940a9ac0 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf9be2e640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf9be2e640 4 set_mon_vals no callback set 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 1 -- 172.21.15.37:0/1232031157 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbf90001cb0 con 0x7fbf940a9ac0 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.472+0000 7fbf98bc8640 10 monclient: dump: 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:14.480 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:14.481 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:14.481 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:14.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.473+0000 7fbf9be2e640 10 monclient: shutdown 2024-03-29T11:44:14.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.473+0000 7fbf9be2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:14.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.473+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1232031157 >> v1:172.21.15.37:6790/0 conn(0x7fbf940a9ac0 legacy=0x7fbf940adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:14.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.473+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1232031157 shutdown_connections 2024-03-29T11:44:14.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.473+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1232031157 >> 172.21.15.37:0/1232031157 conn(0x7fbf940b35e0 msgr2=0x7fbf940b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:14.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.476+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1232031157 shutdown_connections 2024-03-29T11:44:14.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.477+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1232031157 wait complete. 2024-03-29T11:44:14.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.478+0000 7fbf9be2e640 10 monclient: build_initial_monmap 2024-03-29T11:44:14.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.478+0000 7fbf9be2e640 10 monclient: monmap: 2024-03-29T11:44:14.483 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:14.483 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:14.483 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:44:14.483 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:44:14.483 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:14.483 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:14.483 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:14.483 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:44:14.483 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:44:14.484 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:14.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.478+0000 7fbf9be2e640 1 librados: starting msgr at 2024-03-29T11:44:14.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.478+0000 7fbf9be2e640 1 librados: starting objecter 2024-03-29T11:44:14.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.479+0000 7fbf9be2e640 1 Processor -- start 2024-03-29T11:44:14.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.479+0000 7fbf9be2e640 1 -- start start 2024-03-29T11:44:14.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.479+0000 7fbf9be2e640 1 librados: setting wanted keys 2024-03-29T11:44:14.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.479+0000 7fbf9be2e640 1 librados: calling monclient init 2024-03-29T11:44:14.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.479+0000 7fbf9be2e640 10 monclient: init 2024-03-29T11:44:14.485 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.480+0000 7fbf9be2e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:14.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.480+0000 7fbf9be2e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:14.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.480+0000 7fbf9be2e640 10 monclient(hunting): picked mon.noname-b con 0x7fbf940a9ac0 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:14.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.480+0000 7fbf9be2e640 10 monclient(hunting): picked mon.noname-c con 0x7fbf940af900 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:14.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.480+0000 7fbf9be2e640 10 monclient(hunting): picked mon.noname-a con 0x7fbf9413ee90 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:14.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.480+0000 7fbf9be2e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9413c1f0 con 0x7fbf9413ee90 2024-03-29T11:44:14.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.480+0000 7fbf9be2e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9414ca60 con 0x7fbf940a9ac0 2024-03-29T11:44:14.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.480+0000 7fbf9be2e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9414cbf0 con 0x7fbf940af900 2024-03-29T11:44:14.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.480+0000 7fbf9be2e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:14.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.480+0000 7fbf9be2e640 10 monclient(hunting): authenticate will time out at 1433.929098s 2024-03-29T11:44:14.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.481+0000 7fbf993c9640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fbf940af900 0x7fbf941d79e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:59974/0 (socket says 172.21.15.37:59974) 2024-03-29T11:44:14.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.481+0000 7fbf993c9640 1 -- 172.21.15.37:0/1633870105 learned_addr learned my addr 172.21.15.37:0/1633870105 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:14.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.481+0000 7fbf8a7fc640 10 client.?.objecter ms_handle_connect 0x7fbf940af900 2024-03-29T11:44:14.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.481+0000 7fbf8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:14.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.485+0000 7fbf8a7fc640 10 client.?.objecter ms_handle_connect 0x7fbf9413ee90 2024-03-29T11:44:14.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.485+0000 7fbf8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:14.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.486+0000 7fbf8a7fc640 10 client.?.objecter ms_handle_connect 0x7fbf940a9ac0 2024-03-29T11:44:14.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.486+0000 7fbf8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:14.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.486+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977524228 0 0) 0x7fbf9413c1f0 con 0x7fbf9413ee90 2024-03-29T11:44:14.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.486+0000 7fbf8a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:14.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.486+0000 7fbf8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:14.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.486+0000 7fbf8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:14.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.486+0000 7fbf8a7fc640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:44:14.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.486+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf78004eb0 con 0x7fbf9413ee90 2024-03-29T11:44:14.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.487+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1279751399 0 0) 0x7fbf78004eb0 con 0x7fbf9413ee90 2024-03-29T11:44:14.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.487+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf78005300 con 0x7fbf9413ee90 2024-03-29T11:44:14.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.487+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbf840017b0 con 0x7fbf9413ee90 2024-03-29T11:44:14.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.487+0000 7fbf8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:14.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.487+0000 7fbf8a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:14.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.487+0000 7fbf8a7fc640 10 monclient(hunting): dump: 2024-03-29T11:44:14.492 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:14.492 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:14.493 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:14.493 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:14.493 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:14.493 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:14.493 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:14.493 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:14.493 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:14.493 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:14.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.487+0000 7fbf8a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:14.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.487+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451741891 0 0) 0x7fbf9414ca60 con 0x7fbf940a9ac0 2024-03-29T11:44:14.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.487+0000 7fbf8a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:14.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.487+0000 7fbf8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:14.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.487+0000 7fbf8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:14.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.487+0000 7fbf8a7fc640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:44:14.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.487+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf78004eb0 con 0x7fbf940a9ac0 2024-03-29T11:44:14.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2902489318 0 0) 0x7fbf78005300 con 0x7fbf9413ee90 2024-03-29T11:44:14.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 >> v1:172.21.15.37:6790/0 conn(0x7fbf940af900 legacy=0x7fbf941d79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:14.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 >> v1:172.21.15.146:6789/0 conn(0x7fbf940a9ac0 legacy=0x7fbf941286c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:14.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf8a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:14.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf8a7fc640 1 monclient: found mon.a 2024-03-29T11:44:14.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:14.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf8a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:14.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf94128dd0 con 0x7fbf9413ee90 2024-03-29T11:44:14.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf8a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:14.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf8a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:14.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf9be2e640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf9be2e640 10 monclient: _renew_subs 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf9be2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1633870105 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf94129080 con 0x7fbf9413ee90 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf9be2e640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf9be2e640 10 monclient: _renew_subs 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf9be2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1633870105 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf941348a0 con 0x7fbf9413ee90 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.488+0000 7fbf9be2e640 1 librados: init done 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.489+0000 7fbf9be2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.489+0000 7fbf9be2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.489+0000 7fbf9be2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.489+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1633870105 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9413c430 con 0x7fbf9413ee90 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.489+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf84001d00 con 0x7fbf9413ee90 2024-03-29T11:44:14.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.489+0000 7fbf8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:14.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.489+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbf84003790 con 0x7fbf9413ee90 2024-03-29T11:44:14.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.489+0000 7fbf8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:14.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.489+0000 7fbf8a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:14.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.489+0000 7fbf8a7fc640 10 monclient: dump: 2024-03-29T11:44:14.496 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:14.496 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:14.496 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:14.496 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:14.496 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:14.496 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:14.496 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:14.496 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:14.496 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:14.496 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:14.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.490+0000 7fbf8bfff640 4 set_mon_vals no callback set 2024-03-29T11:44:14.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.490+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fbf84019760 con 0x7fbf9413ee90 2024-03-29T11:44:14.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.491+0000 7fbf99bca640 10 monclient: get_auth_request con 0x7fbf7806e0f0 auth_method 0 2024-03-29T11:44:14.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.491+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fbf84053310 con 0x7fbf9413ee90 2024-03-29T11:44:14.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.491+0000 7fbf8a7fc640 10 client.4545.objecter ms_dispatch 0x7fbf94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:14.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.491+0000 7fbf8a7fc640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:14.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.491+0000 7fbf8a7fc640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:14.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.491+0000 7fbf8a7fc640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-29T11:44:14.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.492+0000 7fbf8a7fc640 10 client.4545.objecter ms_handle_connect 0x7fbf7806e0f0 2024-03-29T11:44:14.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.500+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf8404f5b0 con 0x7fbf9413ee90 2024-03-29T11:44:14.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.500+0000 7fbf8a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:14.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.500+0000 7fbf8a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:14.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.826+0000 7fbf9be2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:14.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.826+0000 7fbf9be2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:14.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.826+0000 7fbf9be2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:14.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.826+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1633870105 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf9413c3e0 con 0x7fbf9413ee90 2024-03-29T11:44:14.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.827+0000 7fbf8a7fc640 1 -- 172.21.15.37:0/1633870105 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fbf8404f750 con 0x7fbf9413ee90 2024-03-29T11:44:14.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.827+0000 7fbf8a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:14.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.827+0000 7fbf8a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:14.832 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:44:14.832 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:14.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.832+0000 7fbf9be2e640 10 librados: watch_flush enter 2024-03-29T11:44:14.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.832+0000 7fbf9be2e640 10 librados: watch_flush exit 2024-03-29T11:44:14.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.832+0000 7fbf9be2e640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-29T11:44:14.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.832+0000 7fbf9be2e640 10 client.4545.objecter successfully canceled tick 2024-03-29T11:44:14.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.833+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1633870105 >> v1:172.21.15.146:6816/2899149139 conn(0x7fbf7806e0f0 legacy=0x7fbf780705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:14.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.833+0000 7fbf9be2e640 10 monclient: shutdown 2024-03-29T11:44:14.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.833+0000 7fbf9be2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:14.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.833+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1633870105 >> v1:172.21.15.37:6789/0 conn(0x7fbf9413ee90 legacy=0x7fbf941e3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:14.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.833+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1633870105 shutdown_connections 2024-03-29T11:44:14.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.833+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1633870105 >> 172.21.15.37:0/1633870105 conn(0x7fbf940b35e0 msgr2=0x7fbf940b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:14.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.833+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1633870105 shutdown_connections 2024-03-29T11:44:14.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.833+0000 7fbf9be2e640 1 -- 172.21.15.37:0/1633870105 wait complete. 2024-03-29T11:44:14.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:14.834+0000 7fbf9be2e640 1 librados: shutdown 2024-03-29T11:44:14.854 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:21.661 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:21.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.819+0000 7fbb7cf83640 10 monclient: get_monmap_and_config 2024-03-29T11:44:21.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.819+0000 7fbb7cf83640 10 monclient: build_initial_monmap 2024-03-29T11:44:21.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.819+0000 7fbb7cf83640 10 monclient: monmap: 2024-03-29T11:44:21.824 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:21.824 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:21.824 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:44:21.821138+0000 2024-03-29T11:44:21.824 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:44:21.821138+0000 2024-03-29T11:44:21.824 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:21.824 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:21.824 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:21.825 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:44:21.825 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:44:21.825 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:21.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.824+0000 7fbb7cf83640 1 Processor -- start 2024-03-29T11:44:21.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.824+0000 7fbb7cf83640 1 -- start start 2024-03-29T11:44:21.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.824+0000 7fbb7cf83640 10 monclient: init 2024-03-29T11:44:21.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.825+0000 7fbb7cf83640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:21.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.825+0000 7fbb7cf83640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:21.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.825+0000 7fbb7cf83640 10 monclient(hunting): picked mon.noname-a con 0x7fbb781422b0 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:21.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.825+0000 7fbb7cf83640 10 monclient(hunting): picked mon.noname-c con 0x7fbb78142690 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:21.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.825+0000 7fbb7cf83640 10 monclient(hunting): picked mon.noname-b con 0x7fbb780b17d0 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:21.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.825+0000 7fbb7cf83640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb78142c80 con 0x7fbb781422b0 2024-03-29T11:44:21.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.825+0000 7fbb7cf83640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb780aae40 con 0x7fbb78142690 2024-03-29T11:44:21.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.825+0000 7fbb7cf83640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7813fbd0 con 0x7fbb780b17d0 2024-03-29T11:44:21.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.825+0000 7fbb7cf83640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:21.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.825+0000 7fbb7cf83640 10 monclient(hunting): authenticate will time out at 1441.275440s 2024-03-29T11:44:21.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.826+0000 7fbb75d9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbb78142690 0x7fbb780b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:49318/0 (socket says 172.21.15.37:49318) 2024-03-29T11:44:21.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.826+0000 7fbb75d9b640 1 -- 172.21.15.37:0/1504983375 learned_addr learned my addr 172.21.15.37:0/1504983375 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:21.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.828+0000 7fbb7559a640 1 -- 172.21.15.37:0/1504983375 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735378508 0 0) 0x7fbb780aae40 con 0x7fbb78142690 2024-03-29T11:44:21.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.828+0000 7fbb7559a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:21.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.828+0000 7fbb7559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:21.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.828+0000 7fbb7559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:21.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.828+0000 7fbb7559a640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:44:21.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.828+0000 7fbb7559a640 1 -- 172.21.15.37:0/1504983375 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb60003520 con 0x7fbb78142690 2024-03-29T11:44:21.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.829+0000 7fbb7559a640 1 -- 172.21.15.37:0/1504983375 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 57392414 0 0) 0x7fbb60003520 con 0x7fbb78142690 2024-03-29T11:44:21.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.829+0000 7fbb7559a640 1 -- 172.21.15.37:0/1504983375 >> v1:172.21.15.37:6790/0 conn(0x7fbb780b17d0 legacy=0x7fbb780b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7559a640 1 -- 172.21.15.37:0/1504983375 >> v1:172.21.15.37:6789/0 conn(0x7fbb781422b0 legacy=0x7fbb7813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7559a640 10 monclient: _finish_hunting 0 2024-03-29T11:44:21.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7559a640 1 monclient: found mon.noname-c 2024-03-29T11:44:21.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:21.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:44:21.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7559a640 1 -- 172.21.15.37:0/1504983375 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb78151940 con 0x7fbb78142690 2024-03-29T11:44:21.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7559a640 10 monclient: _finish_auth 0 2024-03-29T11:44:21.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7559a640 10 monclient: _check_auth_tickets 2024-03-29T11:44:21.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:21.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7559a640 1 -- 172.21.15.37:0/1504983375 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbb6c001690 con 0x7fbb78142690 2024-03-29T11:44:21.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:21.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:21.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7559a640 10 monclient: dump: 2024-03-29T11:44:21.838 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:21.838 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:21.838 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:21.838 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:21.838 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:21.839 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:21.839 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:21.839 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:21.839 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:21.839 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:21.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7cf83640 5 monclient: authenticate success, global_id 4501 2024-03-29T11:44:21.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.830+0000 7fbb7cf83640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:21.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.831+0000 7fbb7559a640 1 -- 172.21.15.37:0/1504983375 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb6c0017f0 con 0x7fbb78142690 2024-03-29T11:44:21.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.831+0000 7fbb7559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:21.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.831+0000 7fbb7559a640 1 -- 172.21.15.37:0/1504983375 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbb6c001bc0 con 0x7fbb78142690 2024-03-29T11:44:21.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.831+0000 7fbb7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:21.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.831+0000 7fbb7559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:21.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.831+0000 7fbb7559a640 10 monclient: dump: 2024-03-29T11:44:21.839 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:21.840 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:21.840 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:21.840 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:21.840 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:21.840 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:21.840 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:21.840 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:21.841 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:21.841 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:21.841 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.831+0000 7fbb7cf83640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:21.841 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.831+0000 7fbb7cf83640 4 set_mon_vals no callback set 2024-03-29T11:44:21.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.832+0000 7fbb7cf83640 10 monclient: shutdown 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.832+0000 7fbb7cf83640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.832+0000 7fbb7cf83640 1 -- 172.21.15.37:0/1504983375 >> v1:172.21.15.146:6789/0 conn(0x7fbb78142690 legacy=0x7fbb780b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.832+0000 7fbb7cf83640 1 -- 172.21.15.37:0/1504983375 shutdown_connections 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.832+0000 7fbb7cf83640 1 -- 172.21.15.37:0/1504983375 >> 172.21.15.37:0/1504983375 conn(0x7fbb78066a00 msgr2=0x7fbb78066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.834+0000 7fbb7cf83640 1 -- 172.21.15.37:0/1504983375 shutdown_connections 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.834+0000 7fbb7cf83640 1 -- 172.21.15.37:0/1504983375 wait complete. 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.835+0000 7fbb7cf83640 10 monclient: build_initial_monmap 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.835+0000 7fbb7cf83640 10 monclient: monmap: 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:21.844 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.835+0000 7fbb7cf83640 1 librados: starting msgr at 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.835+0000 7fbb7cf83640 1 librados: starting objecter 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.835+0000 7fbb7cf83640 1 Processor -- start 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.835+0000 7fbb7cf83640 1 -- start start 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.835+0000 7fbb7cf83640 1 librados: setting wanted keys 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.835+0000 7fbb7cf83640 1 librados: calling monclient init 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.835+0000 7fbb7cf83640 10 monclient: init 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.836+0000 7fbb7cf83640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.836+0000 7fbb7cf83640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.837+0000 7fbb7cf83640 10 monclient(hunting): picked mon.noname-b con 0x7fbb780b17d0 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.837+0000 7fbb7cf83640 10 monclient(hunting): picked mon.noname-a con 0x7fbb781422b0 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:21.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.838+0000 7fbb7cf83640 10 monclient(hunting): picked mon.noname-c con 0x7fbb78142690 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:21.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.838+0000 7fbb7cf83640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb78132510 con 0x7fbb781422b0 2024-03-29T11:44:21.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.838+0000 7fbb7cf83640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb780ac6e0 con 0x7fbb780b17d0 2024-03-29T11:44:21.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.838+0000 7fbb7cf83640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb780ac930 con 0x7fbb78142690 2024-03-29T11:44:21.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.838+0000 7fbb7cf83640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:21.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.838+0000 7fbb7cf83640 10 monclient(hunting): authenticate will time out at 1441.288385s 2024-03-29T11:44:21.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.838+0000 7fbb7659c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbb780b17d0 0x7fbb780ab7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:38856/0 (socket says 172.21.15.37:38856) 2024-03-29T11:44:21.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.838+0000 7fbb7659c640 1 -- 172.21.15.37:0/2977634267 learned_addr learned my addr 172.21.15.37:0/2977634267 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:21.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.842+0000 7fbb66ffd640 10 client.?.objecter ms_handle_connect 0x7fbb780b17d0 2024-03-29T11:44:21.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.842+0000 7fbb66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:21.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.842+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894927428 0 0) 0x7fbb780ac6e0 con 0x7fbb780b17d0 2024-03-29T11:44:21.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.842+0000 7fbb66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:21.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.842+0000 7fbb66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:21.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.842+0000 7fbb66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:21.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.842+0000 7fbb66ffd640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:44:21.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.842+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb58004eb0 con 0x7fbb780b17d0 2024-03-29T11:44:21.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.842+0000 7fbb66ffd640 10 client.?.objecter ms_handle_connect 0x7fbb781422b0 2024-03-29T11:44:21.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.842+0000 7fbb66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:21.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4255083178 0 0) 0x7fbb58004eb0 con 0x7fbb780b17d0 2024-03-29T11:44:21.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb58005300 con 0x7fbb780b17d0 2024-03-29T11:44:21.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbb680019f0 con 0x7fbb780b17d0 2024-03-29T11:44:21.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:21.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:21.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 10 monclient(hunting): dump: 2024-03-29T11:44:21.850 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:21.850 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:21.851 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:21.851 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:21.853 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:21.853 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:21.853 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:21.853 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:21.854 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:21.854 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:21.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:21.858 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3748984739 0 0) 0x7fbb58005300 con 0x7fbb780b17d0 2024-03-29T11:44:21.858 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 >> v1:172.21.15.37:6790/0 conn(0x7fbb78142690 legacy=0x7fbb78130560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.859 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 >> v1:172.21.15.37:6789/0 conn(0x7fbb781422b0 legacy=0x7fbb7812fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.859 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:21.859 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 1 monclient: found mon.b 2024-03-29T11:44:21.859 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:21.859 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:44:21.859 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb780acc10 con 0x7fbb780b17d0 2024-03-29T11:44:21.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 10 monclient: _finish_auth 0 2024-03-29T11:44:21.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:21.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.845+0000 7fbb66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:21.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb7cf83640 5 monclient: authenticate success, global_id 4504 2024-03-29T11:44:21.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb7cf83640 10 monclient: _renew_subs 2024-03-29T11:44:21.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb7cf83640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:44:21.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb7cf83640 1 -- 172.21.15.37:0/2977634267 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb781320a0 con 0x7fbb780b17d0 2024-03-29T11:44:21.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb7cf83640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:21.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb68001cb0 con 0x7fbb780b17d0 2024-03-29T11:44:21.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:21.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fbb680019f0 con 0x7fbb780b17d0 2024-03-29T11:44:21.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:21.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:21.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb66ffd640 10 monclient: dump: 2024-03-29T11:44:21.861 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:21.861 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:21.861 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:21.861 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:21.861 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:21.861 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:21.862 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:21.862 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:21.862 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:21.862 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:21.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb7cf83640 10 monclient: _renew_subs 2024-03-29T11:44:21.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb7cf83640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:44:21.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb7cf83640 1 -- 172.21.15.37:0/2977634267 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb781e2f50 con 0x7fbb780b17d0 2024-03-29T11:44:21.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb7cf83640 1 librados: init done 2024-03-29T11:44:21.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.846+0000 7fbb74d99640 4 set_mon_vals no callback set 2024-03-29T11:44:21.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.848+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fbb68003940 con 0x7fbb780b17d0 2024-03-29T11:44:21.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.849+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7fbb68054640 con 0x7fbb780b17d0 2024-03-29T11:44:21.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.849+0000 7fbb66ffd640 10 client.4504.objecter ms_dispatch 0x7fbb78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:21.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.849+0000 7fbb66ffd640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:21.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.849+0000 7fbb66ffd640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:21.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.849+0000 7fbb66ffd640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-29T11:44:21.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.849+0000 7fbb75d9b640 10 monclient: get_auth_request con 0x7fbb5806e170 auth_method 0 2024-03-29T11:44:21.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.850+0000 7fbb7cf83640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:21.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.850+0000 7fbb7cf83640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:21.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.850+0000 7fbb7cf83640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:44:21.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.850+0000 7fbb66ffd640 10 client.4504.objecter ms_handle_connect 0x7fbb5806e170 2024-03-29T11:44:21.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.850+0000 7fbb7cf83640 1 -- 172.21.15.37:0/2977634267 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb7812fc30 con 0x7fbb780b17d0 2024-03-29T11:44:21.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.858+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb680508e0 con 0x7fbb780b17d0 2024-03-29T11:44:21.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.858+0000 7fbb66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:21.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:21.858+0000 7fbb66ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:22.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.220+0000 7fbb7cf83640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:22.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.220+0000 7fbb7cf83640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:22.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.220+0000 7fbb7cf83640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:44:22.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.220+0000 7fbb7cf83640 1 -- 172.21.15.37:0/2977634267 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb78075f10 con 0x7fbb780b17d0 2024-03-29T11:44:22.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.224+0000 7fbb66ffd640 1 -- 172.21.15.37:0/2977634267 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7fbb680508e0 con 0x7fbb780b17d0 2024-03-29T11:44:22.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.224+0000 7fbb66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:22.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.224+0000 7fbb66ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:22.230 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:44:22.231 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:22.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.229+0000 7fbb7cf83640 10 librados: watch_flush enter 2024-03-29T11:44:22.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.229+0000 7fbb7cf83640 10 librados: watch_flush exit 2024-03-29T11:44:22.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.229+0000 7fbb7cf83640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-29T11:44:22.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.229+0000 7fbb7cf83640 10 client.4504.objecter successfully canceled tick 2024-03-29T11:44:22.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.230+0000 7fbb7cf83640 1 -- 172.21.15.37:0/2977634267 >> v1:172.21.15.146:6816/2899149139 conn(0x7fbb5806e170 legacy=0x7fbb58070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:22.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.230+0000 7fbb7cf83640 10 monclient: shutdown 2024-03-29T11:44:22.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.230+0000 7fbb7cf83640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:22.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.230+0000 7fbb7cf83640 1 -- 172.21.15.37:0/2977634267 >> v1:172.21.15.146:6789/0 conn(0x7fbb780b17d0 legacy=0x7fbb780ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:22.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.230+0000 7fbb7cf83640 1 -- 172.21.15.37:0/2977634267 shutdown_connections 2024-03-29T11:44:22.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.230+0000 7fbb7cf83640 1 -- 172.21.15.37:0/2977634267 >> 172.21.15.37:0/2977634267 conn(0x7fbb78066a00 msgr2=0x7fbb7813d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:22.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.230+0000 7fbb7cf83640 1 -- 172.21.15.37:0/2977634267 shutdown_connections 2024-03-29T11:44:22.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.230+0000 7fbb7cf83640 1 -- 172.21.15.37:0/2977634267 wait complete. 2024-03-29T11:44:22.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:22.230+0000 7fbb7cf83640 1 librados: shutdown 2024-03-29T11:44:22.257 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:29.066 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:29.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.208+0000 7f1cffcff640 10 monclient: get_monmap_and_config 2024-03-29T11:44:29.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.208+0000 7f1cffcff640 10 monclient: build_initial_monmap 2024-03-29T11:44:29.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.208+0000 7f1cffcff640 10 monclient: monmap: 2024-03-29T11:44:29.214 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:29.214 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:29.214 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:44:29.210824+0000 2024-03-29T11:44:29.214 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:44:29.210824+0000 2024-03-29T11:44:29.214 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:29.215 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:29.215 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:29.215 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:44:29.215 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:44:29.215 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:29.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.209+0000 7f1cffcff640 1 Processor -- start 2024-03-29T11:44:29.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.210+0000 7f1cffcff640 1 -- start start 2024-03-29T11:44:29.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.210+0000 7f1cffcff640 10 monclient: init 2024-03-29T11:44:29.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.210+0000 7f1cffcff640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:29.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cffcff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:29.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cffcff640 10 monclient(hunting): picked mon.noname-c con 0x7f1cf80a9750 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:29.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cffcff640 10 monclient(hunting): picked mon.noname-b con 0x7f1cf813e1e0 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:29.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cffcff640 10 monclient(hunting): picked mon.noname-a con 0x7f1cf8071c30 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:29.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cffcff640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf80ab300 con 0x7f1cf8071c30 2024-03-29T11:44:29.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cffcff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf80ac2f0 con 0x7f1cf80a9750 2024-03-29T11:44:29.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cffcff640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf80ad8d0 con 0x7f1cf813e1e0 2024-03-29T11:44:29.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cffcff640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:29.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cffcff640 10 monclient(hunting): authenticate will time out at 1448.661044s 2024-03-29T11:44:29.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cfd29a640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f1cf813e1e0 0x7f1cf80ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:35460/0 (socket says 172.21.15.37:35460) 2024-03-29T11:44:29.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cfd29a640 1 -- 172.21.15.37:0/2451675356 learned_addr learned my addr 172.21.15.37:0/2451675356 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:29.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cfca99640 1 -- 172.21.15.37:0/2451675356 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793409901 0 0) 0x7f1cf80ad8d0 con 0x7f1cf813e1e0 2024-03-29T11:44:29.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cfca99640 10 monclient(hunting): _negotiate 2024-03-29T11:44:29.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cfca99640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:29.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cfca99640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:29.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cfca99640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:44:29.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.211+0000 7f1cfca99640 1 -- 172.21.15.37:0/2451675356 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce4003520 con 0x7f1cf813e1e0 2024-03-29T11:44:29.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 1 -- 172.21.15.37:0/2451675356 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 940096757 0 0) 0x7f1ce4003520 con 0x7f1cf813e1e0 2024-03-29T11:44:29.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 1 -- 172.21.15.37:0/2451675356 >> v1:172.21.15.146:6789/0 conn(0x7f1cf80a9750 legacy=0x7f1cf813dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:29.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 1 -- 172.21.15.37:0/2451675356 >> v1:172.21.15.37:6789/0 conn(0x7f1cf8071c30 legacy=0x7f1cf80ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:29.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 10 monclient: _finish_hunting 0 2024-03-29T11:44:29.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 1 monclient: found mon.noname-b 2024-03-29T11:44:29.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:29.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:44:29.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 1 -- 172.21.15.37:0/2451675356 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf80ada60 con 0x7f1cf813e1e0 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 10 monclient: _finish_auth 0 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 10 monclient: _check_auth_tickets 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cffcff640 5 monclient: authenticate success, global_id 4616 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cffcff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 1 -- 172.21.15.37:0/2451675356 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1ce8001690 con 0x7f1cf813e1e0 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cfca99640 10 monclient: dump: 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:29.219 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:29.220 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:29.220 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:29.220 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:29.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.212+0000 7f1cffcff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:29.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.213+0000 7f1cfca99640 1 -- 172.21.15.37:0/2451675356 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ce80018b0 con 0x7f1cf813e1e0 2024-03-29T11:44:29.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.213+0000 7f1cfca99640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:29.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.213+0000 7f1cffcff640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:29.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.213+0000 7f1cffcff640 4 set_mon_vals no callback set 2024-03-29T11:44:29.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.213+0000 7f1cfca99640 1 -- 172.21.15.37:0/2451675356 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1ce8001c40 con 0x7f1cf813e1e0 2024-03-29T11:44:29.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.213+0000 7f1cfca99640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:29.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.214+0000 7f1cfca99640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:29.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.214+0000 7f1cfca99640 10 monclient: dump: 2024-03-29T11:44:29.226 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.214+0000 7f1cffcff640 10 monclient: shutdown 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.214+0000 7f1cffcff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.214+0000 7f1cffcff640 1 -- 172.21.15.37:0/2451675356 >> v1:172.21.15.37:6790/0 conn(0x7f1cf813e1e0 legacy=0x7f1cf80ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.214+0000 7f1cffcff640 1 -- 172.21.15.37:0/2451675356 shutdown_connections 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.214+0000 7f1cffcff640 1 -- 172.21.15.37:0/2451675356 >> 172.21.15.37:0/2451675356 conn(0x7f1cf806b660 msgr2=0x7f1cf806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.214+0000 7f1cffcff640 1 -- 172.21.15.37:0/2451675356 shutdown_connections 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.214+0000 7f1cffcff640 1 -- 172.21.15.37:0/2451675356 wait complete. 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.215+0000 7f1cffcff640 10 monclient: build_initial_monmap 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.215+0000 7f1cffcff640 10 monclient: monmap: 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.215+0000 7f1cffcff640 1 librados: starting msgr at 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.215+0000 7f1cffcff640 1 librados: starting objecter 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.215+0000 7f1cffcff640 1 Processor -- start 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.216+0000 7f1cffcff640 1 -- start start 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.216+0000 7f1cffcff640 1 librados: setting wanted keys 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.216+0000 7f1cffcff640 1 librados: calling monclient init 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.216+0000 7f1cffcff640 10 monclient: init 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.216+0000 7f1cffcff640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.216+0000 7f1cffcff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.216+0000 7f1cffcff640 10 monclient(hunting): picked mon.noname-a con 0x7f1cf8071c30 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.216+0000 7f1cffcff640 10 monclient(hunting): picked mon.noname-b con 0x7f1cf80a9750 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.216+0000 7f1cffcff640 10 monclient(hunting): picked mon.noname-c con 0x7f1cf812d1e0 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.216+0000 7f1cffcff640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf8070430 con 0x7f1cf8071c30 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.216+0000 7f1cffcff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf811b1c0 con 0x7f1cf80a9750 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.216+0000 7f1cffcff640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf8070660 con 0x7f1cf812d1e0 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.216+0000 7f1cffcff640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.216+0000 7f1cffcff640 10 monclient(hunting): authenticate will time out at 1448.666865s 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.217+0000 7f1cfda9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f1cf8071c30 0x7f1cf806f060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36524/0 (socket says 172.21.15.37:36524) 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.217+0000 7f1cfda9b640 1 -- 172.21.15.37:0/1668148437 learned_addr learned my addr 172.21.15.37:0/1668148437 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.217+0000 7f1cee7fc640 10 client.?.objecter ms_handle_connect 0x7f1cf8071c30 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.217+0000 7f1cee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.217+0000 7f1cee7fc640 10 client.?.objecter ms_handle_connect 0x7f1cf812d1e0 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.217+0000 7f1cee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819542503 0 0) 0x7f1cf8070660 con 0x7f1cf812d1e0 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cdc004eb0 con 0x7f1cf812d1e0 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901881397 0 0) 0x7f1cf8070430 con 0x7f1cf8071c30 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf8070660 con 0x7f1cf8071c30 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2764007851 0 0) 0x7f1cf8070660 con 0x7f1cf8071c30 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cdc0054e0 con 0x7f1cf8071c30 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1cf0001c90 con 0x7f1cf8071c30 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 10 monclient(hunting): dump: 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:29.232 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.218+0000 7f1cee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 204122587 0 0) 0x7f1cdc004eb0 con 0x7f1cf812d1e0 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cf8070660 con 0x7f1cf812d1e0 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1cf4001760 con 0x7f1cf812d1e0 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 10 monclient(hunting): dump: 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 10 client.?.objecter ms_handle_connect 0x7f1cf80a9750 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2599447644 0 0) 0x7f1cdc0054e0 con 0x7f1cf8071c30 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 >> v1:172.21.15.37:6790/0 conn(0x7f1cf812d1e0 legacy=0x7f1cf81dffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 >> v1:172.21.15.146:6789/0 conn(0x7f1cf80a9750 legacy=0x7f1cf8071040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 1 monclient: found mon.a 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf8070940 con 0x7f1cf8071c30 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cffcff640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cffcff640 10 monclient: _renew_subs 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cffcff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cffcff640 1 -- 172.21.15.37:0/1668148437 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf8070c20 con 0x7f1cf8071c30 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cffcff640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cffcff640 10 monclient: _renew_subs 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cffcff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:29.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cffcff640 1 -- 172.21.15.37:0/1668148437 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf8129f50 con 0x7f1cf8071c30 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.219+0000 7f1cffcff640 1 librados: init done 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.220+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cf0003990 con 0x7f1cf8071c30 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.220+0000 7f1cee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.220+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1cf0003d20 con 0x7f1cf8071c30 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.220+0000 7f1cee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.220+0000 7f1ceffff640 4 set_mon_vals no callback set 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.220+0000 7f1cee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.220+0000 7f1cee7fc640 10 monclient: dump: 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:29.236 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:29.237 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:29.237 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:29.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.221+0000 7f1cffcff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:29.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.221+0000 7f1cffcff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:29.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.221+0000 7f1cffcff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:29.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.222+0000 7f1cffcff640 1 -- 172.21.15.37:0/1668148437 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce0005080 con 0x7f1cf8071c30 2024-03-29T11:44:29.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.224+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f1cf0003f00 con 0x7f1cf8071c30 2024-03-29T11:44:29.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.224+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f1cf0084b70 con 0x7f1cf8071c30 2024-03-29T11:44:29.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.224+0000 7f1cee7fc640 10 client.4554.objecter ms_dispatch 0x7f1cf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:29.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.224+0000 7f1cee7fc640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:29.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.224+0000 7f1cee7fc640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:29.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.224+0000 7f1cee7fc640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-29T11:44:29.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.224+0000 7f1cfd29a640 10 monclient: get_auth_request con 0x7f1cdc06e2e0 auth_method 0 2024-03-29T11:44:29.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.225+0000 7f1cee7fc640 10 client.4554.objecter ms_handle_connect 0x7f1cdc06e2e0 2024-03-29T11:44:29.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.230+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1cf0050280 con 0x7f1cf8071c30 2024-03-29T11:44:29.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.230+0000 7f1cee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:29.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.230+0000 7f1cee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:29.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.615+0000 7f1cffcff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:29.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.615+0000 7f1cffcff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:29.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.615+0000 7f1cffcff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:29.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.615+0000 7f1cffcff640 1 -- 172.21.15.37:0/1668148437 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ce0005330 con 0x7f1cf8071c30 2024-03-29T11:44:29.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.616+0000 7f1cee7fc640 1 -- 172.21.15.37:0/1668148437 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f1cf0003b80 con 0x7f1cf8071c30 2024-03-29T11:44:29.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.616+0000 7f1cee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:29.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.616+0000 7f1cee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:29.622 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:44:29.622 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:29.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.621+0000 7f1cffcff640 10 librados: watch_flush enter 2024-03-29T11:44:29.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.621+0000 7f1cffcff640 10 librados: watch_flush exit 2024-03-29T11:44:29.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.621+0000 7f1cffcff640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-29T11:44:29.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.621+0000 7f1cffcff640 10 client.4554.objecter successfully canceled tick 2024-03-29T11:44:29.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.621+0000 7f1cffcff640 1 -- 172.21.15.37:0/1668148437 >> v1:172.21.15.146:6816/2899149139 conn(0x7f1cdc06e2e0 legacy=0x7f1cdc0707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:29.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.621+0000 7f1cffcff640 10 monclient: shutdown 2024-03-29T11:44:29.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.621+0000 7f1cffcff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:29.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.621+0000 7f1cffcff640 1 -- 172.21.15.37:0/1668148437 >> v1:172.21.15.37:6789/0 conn(0x7f1cf8071c30 legacy=0x7f1cf806f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:29.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.621+0000 7f1cffcff640 1 -- 172.21.15.37:0/1668148437 shutdown_connections 2024-03-29T11:44:29.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.621+0000 7f1cffcff640 1 -- 172.21.15.37:0/1668148437 >> 172.21.15.37:0/1668148437 conn(0x7f1cf806b660 msgr2=0x7f1cf812fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:29.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.621+0000 7f1cffcff640 1 -- 172.21.15.37:0/1668148437 shutdown_connections 2024-03-29T11:44:29.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.621+0000 7f1cffcff640 1 -- 172.21.15.37:0/1668148437 wait complete. 2024-03-29T11:44:29.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:29.621+0000 7f1cffcff640 1 librados: shutdown 2024-03-29T11:44:29.642 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:36.448 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:36.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.594+0000 7f6871f65640 10 monclient: get_monmap_and_config 2024-03-29T11:44:36.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.594+0000 7f6871f65640 10 monclient: build_initial_monmap 2024-03-29T11:44:36.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.594+0000 7f6871f65640 10 monclient: monmap: 2024-03-29T11:44:36.599 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:36.599 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:36.599 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:44:36.595570+0000 2024-03-29T11:44:36.599 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:44:36.595570+0000 2024-03-29T11:44:36.599 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:36.599 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:36.599 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:36.599 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:44:36.599 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:44:36.599 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:36.600 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.596+0000 7f6871f65640 1 Processor -- start 2024-03-29T11:44:36.600 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.596+0000 7f6871f65640 1 -- start start 2024-03-29T11:44:36.600 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.596+0000 7f6871f65640 10 monclient: init 2024-03-29T11:44:36.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.597+0000 7f6871f65640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:36.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.597+0000 7f6871f65640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:36.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.599+0000 7f6871f65640 10 monclient(hunting): picked mon.noname-b con 0x7f686c0af390 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:36.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.600+0000 7f686b7fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f686c0af390 0x7f686c0b1030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:57033/0 (socket says 172.21.15.37:57033) 2024-03-29T11:44:36.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.600+0000 7f686b7fe640 1 -- 172.21.15.37:0/831992465 learned_addr learned my addr 172.21.15.37:0/831992465 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:36.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.600+0000 7f6871f65640 10 monclient(hunting): picked mon.noname-a con 0x7f686c0b4ab0 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:36.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.600+0000 7f6871f65640 10 monclient(hunting): picked mon.noname-c con 0x7f686c0b1740 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:36.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.600+0000 7f6871f65640 1 -- 172.21.15.37:0/831992465 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c146190 con 0x7f686c0b4ab0 2024-03-29T11:44:36.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.600+0000 7f6871f65640 1 -- 172.21.15.37:0/831992465 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c0b45d0 con 0x7f686c0b1740 2024-03-29T11:44:36.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.600+0000 7f6871f65640 1 -- 172.21.15.37:0/831992465 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c0b0370 con 0x7f686c0af390 2024-03-29T11:44:36.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.600+0000 7f6871f65640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:36.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.600+0000 7f6871f65640 10 monclient(hunting): authenticate will time out at 1456.049594s 2024-03-29T11:44:36.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.601+0000 7f686a7fc640 1 -- 172.21.15.37:0/831992465 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947201796 0 0) 0x7f686c0b0370 con 0x7f686c0af390 2024-03-29T11:44:36.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.601+0000 7f686a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:36.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.601+0000 7f686a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:36.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.601+0000 7f686a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:36.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.601+0000 7f686a7fc640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:44:36.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.601+0000 7f686a7fc640 1 -- 172.21.15.37:0/831992465 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6854003520 con 0x7f686c0af390 2024-03-29T11:44:36.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.601+0000 7f686a7fc640 1 -- 172.21.15.37:0/831992465 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2108003147 0 0) 0x7f6854003520 con 0x7f686c0af390 2024-03-29T11:44:36.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.602+0000 7f686a7fc640 1 -- 172.21.15.37:0/831992465 >> v1:172.21.15.146:6789/0 conn(0x7f686c0b1740 legacy=0x7f686c145a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:36.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.603+0000 7f686a7fc640 1 -- 172.21.15.37:0/831992465 >> v1:172.21.15.37:6789/0 conn(0x7f686c0b4ab0 legacy=0x7f686c145370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:36.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:36.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 1 -- 172.21.15.37:0/831992465 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686c0b0650 con 0x7f686c0af390 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 1 -- 172.21.15.37:0/831992465 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6864001690 con 0x7f686c0af390 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 10 monclient: dump: 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:36.612 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f6871f65640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f6871f65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 1 -- 172.21.15.37:0/831992465 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68640017f0 con 0x7f686c0af390 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 1 -- 172.21.15.37:0/831992465 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f6864001c40 con 0x7f686c0af390 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f686a7fc640 10 monclient: dump: 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:36.613 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:36.614 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:36.614 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:36.614 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:36.614 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:36.614 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:36.614 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:36.614 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:36.614 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:36.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f6871f65640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:36.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.606+0000 7f6871f65640 4 set_mon_vals no callback set 2024-03-29T11:44:36.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.607+0000 7f6871f65640 10 monclient: shutdown 2024-03-29T11:44:36.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.608+0000 7f6871f65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:36.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.608+0000 7f6871f65640 1 -- 172.21.15.37:0/831992465 >> v1:172.21.15.37:6790/0 conn(0x7f686c0af390 legacy=0x7f686c0b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.610+0000 7f6871f65640 1 -- 172.21.15.37:0/831992465 shutdown_connections 2024-03-29T11:44:36.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.610+0000 7f6871f65640 1 -- 172.21.15.37:0/831992465 >> 172.21.15.37:0/831992465 conn(0x7f686c066a00 msgr2=0x7f686c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:36.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.611+0000 7f6871f65640 1 -- 172.21.15.37:0/831992465 shutdown_connections 2024-03-29T11:44:36.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.611+0000 7f6871f65640 1 -- 172.21.15.37:0/831992465 wait complete. 2024-03-29T11:44:36.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.612+0000 7f6871f65640 10 monclient: build_initial_monmap 2024-03-29T11:44:36.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.612+0000 7f6871f65640 10 monclient: monmap: 2024-03-29T11:44:36.618 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:36.618 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:36.618 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:44:36.618 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:44:36.618 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:36.618 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:36.618 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:36.619 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:44:36.619 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:44:36.619 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:36.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.613+0000 7f6871f65640 1 librados: starting msgr at 2024-03-29T11:44:36.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.613+0000 7f6871f65640 1 librados: starting objecter 2024-03-29T11:44:36.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.613+0000 7f6871f65640 1 Processor -- start 2024-03-29T11:44:36.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.613+0000 7f6871f65640 1 -- start start 2024-03-29T11:44:36.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.613+0000 7f6871f65640 1 librados: setting wanted keys 2024-03-29T11:44:36.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.613+0000 7f6871f65640 1 librados: calling monclient init 2024-03-29T11:44:36.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.613+0000 7f6871f65640 10 monclient: init 2024-03-29T11:44:36.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.614+0000 7f6871f65640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:36.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.614+0000 7f6871f65640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:36.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.617+0000 7f6871f65640 10 monclient(hunting): picked mon.noname-a con 0x7f686c0af390 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:36.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.617+0000 7f6871f65640 10 monclient(hunting): picked mon.noname-b con 0x7f686c0b1740 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:36.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.617+0000 7f6871f65640 10 monclient(hunting): picked mon.noname-c con 0x7f686c0b4ab0 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:36.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.617+0000 7f6871f65640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c145140 con 0x7f686c0af390 2024-03-29T11:44:36.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.617+0000 7f6871f65640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c144050 con 0x7f686c0b1740 2024-03-29T11:44:36.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.617+0000 7f6871f65640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c144330 con 0x7f686c0b4ab0 2024-03-29T11:44:36.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.617+0000 7f6871f65640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:36.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.617+0000 7f6871f65640 10 monclient(hunting): authenticate will time out at 1456.066496s 2024-03-29T11:44:36.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.621+0000 7f686affd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f686c0b1740 0x7f686c142500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:46781/0 (socket says 172.21.15.37:46781) 2024-03-29T11:44:36.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.621+0000 7f686affd640 1 -- 172.21.15.37:0/3821913182 learned_addr learned my addr 172.21.15.37:0/3821913182 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:36.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.621+0000 7f6853fff640 10 client.?.objecter ms_handle_connect 0x7f686c0b1740 2024-03-29T11:44:36.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.621+0000 7f6853fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:36.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.621+0000 7f6853fff640 10 client.?.objecter ms_handle_connect 0x7f686c0b4ab0 2024-03-29T11:44:36.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.621+0000 7f6853fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:36.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.622+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251901309 0 0) 0x7f686c144050 con 0x7f686c0b1740 2024-03-29T11:44:36.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.622+0000 7f6853fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:36.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.622+0000 7f6853fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:36.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.622+0000 7f6853fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:36.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.622+0000 7f6853fff640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.622+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6858004eb0 con 0x7f686c0b1740 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.622+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1161235577 0 0) 0x7f6858004eb0 con 0x7f686c0b1740 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.622+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6858005300 con 0x7f686c0b1740 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.623+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f68600017b0 con 0x7f686c0b1740 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.623+0000 7f6853fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.623+0000 7f6853fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.623+0000 7f6853fff640 10 monclient(hunting): dump: 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:36.628 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:36.629 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:36.629 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:36.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.623+0000 7f6853fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:36.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6853fff640 10 client.?.objecter ms_handle_connect 0x7f686c0af390 2024-03-29T11:44:36.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6853fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:36.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3956689686 0 0) 0x7f6858005300 con 0x7f686c0b1740 2024-03-29T11:44:36.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 >> v1:172.21.15.37:6790/0 conn(0x7f686c0b4ab0 legacy=0x7f686c142c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 >> v1:172.21.15.37:6789/0 conn(0x7f686c0af390 legacy=0x7f686c070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6853fff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:36.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6853fff640 1 monclient: found mon.b 2024-03-29T11:44:36.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6853fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:36.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6853fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:44:36.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686c144610 con 0x7f686c0b1740 2024-03-29T11:44:36.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6853fff640 10 monclient: _finish_auth 0 2024-03-29T11:44:36.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6853fff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:36.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6853fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:36.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6871f65640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:44:36.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6871f65640 10 monclient: _renew_subs 2024-03-29T11:44:36.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6871f65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:44:36.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6871f65640 1 -- 172.21.15.37:0/3821913182 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f686c144d10 con 0x7f686c0b1740 2024-03-29T11:44:36.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6871f65640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:36.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6871f65640 10 monclient: _renew_subs 2024-03-29T11:44:36.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6871f65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:44:36.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6871f65640 1 -- 172.21.15.37:0/3821913182 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f686c143d70 con 0x7f686c0b1740 2024-03-29T11:44:36.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.624+0000 7f6871f65640 1 librados: init done 2024-03-29T11:44:36.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.625+0000 7f6871f65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:36.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.625+0000 7f6871f65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:36.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.625+0000 7f6871f65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:44:36.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.625+0000 7f6871f65640 1 -- 172.21.15.37:0/3821913182 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f686c0b55e0 con 0x7f686c0b1740 2024-03-29T11:44:36.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.629+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6860001d40 con 0x7f686c0b1740 2024-03-29T11:44:36.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.629+0000 7f6853fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:36.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.629+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f68600017b0 con 0x7f686c0b1740 2024-03-29T11:44:36.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.629+0000 7f6853fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:36.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.629+0000 7f6853fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:36.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.629+0000 7f68697fa640 4 set_mon_vals no callback set 2024-03-29T11:44:36.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.629+0000 7f6853fff640 10 monclient: dump: 2024-03-29T11:44:36.634 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:36.634 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:36.634 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:36.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.631+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f686001a760 con 0x7f686c0b1740 2024-03-29T11:44:36.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.632+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f6860084650 con 0x7f686c0b1740 2024-03-29T11:44:36.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.632+0000 7f6853fff640 10 client.4513.objecter ms_dispatch 0x7f686c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:36.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.632+0000 7f6853fff640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:36.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.632+0000 7f6853fff640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:36.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.632+0000 7f6853fff640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-29T11:44:36.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.632+0000 7f686b7fe640 10 monclient: get_auth_request con 0x7f685806e030 auth_method 0 2024-03-29T11:44:36.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.633+0000 7f6853fff640 10 client.4513.objecter ms_handle_connect 0x7f685806e030 2024-03-29T11:44:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.639+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f686004fd60 con 0x7f686c0b1740 2024-03-29T11:44:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.639+0000 7f6853fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:36.639+0000 7f6853fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:37.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.019+0000 7f6871f65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:37.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.019+0000 7f6871f65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:37.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.019+0000 7f6871f65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:44:37.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.019+0000 7f6871f65640 1 -- 172.21.15.37:0/3821913182 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f686c144ea0 con 0x7f686c0b1740 2024-03-29T11:44:37.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.020+0000 7f6853fff640 1 -- 172.21.15.37:0/3821913182 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f6860051e60 con 0x7f686c0b1740 2024-03-29T11:44:37.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.020+0000 7f6853fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:37.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.020+0000 7f6853fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:37.024 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:44:37.024 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:37.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.024+0000 7f6871f65640 10 librados: watch_flush enter 2024-03-29T11:44:37.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.024+0000 7f6871f65640 10 librados: watch_flush exit 2024-03-29T11:44:37.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.024+0000 7f6871f65640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-29T11:44:37.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.024+0000 7f6871f65640 10 client.4513.objecter successfully canceled tick 2024-03-29T11:44:37.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.024+0000 7f6871f65640 1 -- 172.21.15.37:0/3821913182 >> v1:172.21.15.146:6816/2899149139 conn(0x7f685806e030 legacy=0x7f68580704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:37.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.024+0000 7f6871f65640 10 monclient: shutdown 2024-03-29T11:44:37.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.024+0000 7f6871f65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:37.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.024+0000 7f6871f65640 1 -- 172.21.15.37:0/3821913182 >> v1:172.21.15.146:6789/0 conn(0x7f686c0b1740 legacy=0x7f686c142500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:37.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.024+0000 7f6871f65640 1 -- 172.21.15.37:0/3821913182 shutdown_connections 2024-03-29T11:44:37.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.024+0000 7f6871f65640 1 -- 172.21.15.37:0/3821913182 >> 172.21.15.37:0/3821913182 conn(0x7f686c066a00 msgr2=0x7f686c145a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:37.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.025+0000 7f6871f65640 1 -- 172.21.15.37:0/3821913182 shutdown_connections 2024-03-29T11:44:37.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.025+0000 7f6871f65640 1 -- 172.21.15.37:0/3821913182 wait complete. 2024-03-29T11:44:37.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:37.025+0000 7f6871f65640 1 librados: shutdown 2024-03-29T11:44:37.043 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:43.850 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:43.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.990+0000 7f059236b640 10 monclient: get_monmap_and_config 2024-03-29T11:44:43.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.990+0000 7f059236b640 10 monclient: build_initial_monmap 2024-03-29T11:44:43.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.990+0000 7f059236b640 10 monclient: monmap: 2024-03-29T11:44:43.995 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:43.995 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:43.995 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:44:43.991791+0000 2024-03-29T11:44:43.995 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:44:43.991791+0000 2024-03-29T11:44:43.995 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:43.995 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:43.995 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:43.996 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:44:43.996 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:44:43.996 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:43.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.992+0000 7f059236b640 1 Processor -- start 2024-03-29T11:44:43.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.992+0000 7f059236b640 1 -- start start 2024-03-29T11:44:43.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.992+0000 7f059236b640 10 monclient: init 2024-03-29T11:44:43.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.992+0000 7f059236b640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:43.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.992+0000 7f059236b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:43.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.992+0000 7f059236b640 10 monclient(hunting): picked mon.noname-b con 0x7f058c0b4330 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:43.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.993+0000 7f058bfff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f058c0b4330 0x7f058c0b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:58242/0 (socket says 172.21.15.37:58242) 2024-03-29T11:44:43.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.993+0000 7f058bfff640 1 -- 172.21.15.37:0/2873249308 learned_addr learned my addr 172.21.15.37:0/2873249308 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:43.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.993+0000 7f059236b640 10 monclient(hunting): picked mon.noname-c con 0x7f058c1470b0 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:44.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.995+0000 7f059236b640 10 monclient(hunting): picked mon.noname-a con 0x7f058c1439b0 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:44.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.995+0000 7f059236b640 1 -- 172.21.15.37:0/2873249308 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c06f2a0 con 0x7f058c1439b0 2024-03-29T11:44:44.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.995+0000 7f059236b640 1 -- 172.21.15.37:0/2873249308 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c146b90 con 0x7f058c1470b0 2024-03-29T11:44:44.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.995+0000 7f059236b640 1 -- 172.21.15.37:0/2873249308 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c0b2ca0 con 0x7f058c0b4330 2024-03-29T11:44:44.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.995+0000 7f059236b640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:44.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.995+0000 7f059236b640 10 monclient(hunting): authenticate will time out at 1463.444151s 2024-03-29T11:44:44.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.995+0000 7f058affd640 1 -- 172.21.15.37:0/2873249308 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976854523 0 0) 0x7f058c0b2ca0 con 0x7f058c0b4330 2024-03-29T11:44:44.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.995+0000 7f058affd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:44.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.995+0000 7f058affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:44.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.995+0000 7f058affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:44.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.995+0000 7f058affd640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:44:44.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.995+0000 7f058affd640 1 -- 172.21.15.37:0/2873249308 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0574003520 con 0x7f058c0b4330 2024-03-29T11:44:44.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 1 -- 172.21.15.37:0/2873249308 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443320935 0 0) 0x7f058c06f2a0 con 0x7f058c1439b0 2024-03-29T11:44:44.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:44.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:44.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:44.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:44:44.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 1 -- 172.21.15.37:0/2873249308 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f058c0b2ca0 con 0x7f058c1439b0 2024-03-29T11:44:44.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 1 -- 172.21.15.37:0/2873249308 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3220981929 0 0) 0x7f0574003520 con 0x7f058c0b4330 2024-03-29T11:44:44.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 1 -- 172.21.15.37:0/2873249308 >> v1:172.21.15.146:6789/0 conn(0x7f058c1470b0 legacy=0x7f058c0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:44.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 1 -- 172.21.15.37:0/2873249308 >> v1:172.21.15.37:6789/0 conn(0x7f058c1439b0 legacy=0x7f058c151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 1 monclient: found mon.noname-b 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 1 -- 172.21.15.37:0/2873249308 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c06e0c0 con 0x7f058c0b4330 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient: _finish_auth 0 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 1 -- 172.21.15.37:0/2873249308 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f057c001690 con 0x7f058c0b4330 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient: dump: 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:44.007 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 1 -- 172.21.15.37:0/2873249308 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f057c0017f0 con 0x7f058c0b4330 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 1 -- 172.21.15.37:0/2873249308 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f057c001c40 con 0x7f058c0b4330 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.996+0000 7f058affd640 10 monclient: dump: 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:44.008 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.998+0000 7f059236b640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.998+0000 7f059236b640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.998+0000 7f059236b640 4 set_mon_vals no callback set 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.998+0000 7f059236b640 10 monclient: shutdown 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.998+0000 7f059236b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.998+0000 7f059236b640 1 -- 172.21.15.37:0/2873249308 >> v1:172.21.15.37:6790/0 conn(0x7f058c0b4330 legacy=0x7f058c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.999+0000 7f059236b640 1 -- 172.21.15.37:0/2873249308 shutdown_connections 2024-03-29T11:44:44.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.999+0000 7f059236b640 1 -- 172.21.15.37:0/2873249308 >> 172.21.15.37:0/2873249308 conn(0x7f058c066a00 msgr2=0x7f058c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.999+0000 7f059236b640 1 -- 172.21.15.37:0/2873249308 shutdown_connections 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:43.999+0000 7f059236b640 1 -- 172.21.15.37:0/2873249308 wait complete. 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.000+0000 7f059236b640 10 monclient: build_initial_monmap 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.000+0000 7f059236b640 10 monclient: monmap: 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:44.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.000+0000 7f059236b640 1 librados: starting msgr at 2024-03-29T11:44:44.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.000+0000 7f059236b640 1 librados: starting objecter 2024-03-29T11:44:44.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.000+0000 7f059236b640 1 Processor -- start 2024-03-29T11:44:44.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.001+0000 7f059236b640 1 -- start start 2024-03-29T11:44:44.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.001+0000 7f059236b640 1 librados: setting wanted keys 2024-03-29T11:44:44.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.001+0000 7f059236b640 1 librados: calling monclient init 2024-03-29T11:44:44.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.001+0000 7f059236b640 10 monclient: init 2024-03-29T11:44:44.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.002+0000 7f059236b640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:44.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.002+0000 7f059236b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:44.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.003+0000 7f059236b640 10 monclient(hunting): picked mon.noname-b con 0x7f058c0b4330 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.003+0000 7f059236b640 10 monclient(hunting): picked mon.noname-a con 0x7f058c1439b0 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.003+0000 7f059236b640 10 monclient(hunting): picked mon.noname-c con 0x7f058c1470b0 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.003+0000 7f059236b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c141460 con 0x7f058c1439b0 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.003+0000 7f059236b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c141730 con 0x7f058c0b4330 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.003+0000 7f059236b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c144350 con 0x7f058c1470b0 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.003+0000 7f059236b640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.003+0000 7f059236b640 10 monclient(hunting): authenticate will time out at 1463.452180s 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.007+0000 7f0590908640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f058c1470b0 0x7f058c140d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:35748/0 (socket says 172.21.15.37:35748) 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.007+0000 7f0590908640 1 -- 172.21.15.37:0/157096349 learned_addr learned my addr 172.21.15.37:0/157096349 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.008+0000 7f0588ff9640 10 client.?.objecter ms_handle_connect 0x7f058c1470b0 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.008+0000 7f0588ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.008+0000 7f0588ff9640 10 client.?.objecter ms_handle_connect 0x7f058c0b4330 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.008+0000 7f0588ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.008+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186475798 0 0) 0x7f058c141730 con 0x7f058c0b4330 2024-03-29T11:44:44.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.008+0000 7f0588ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.008+0000 7f0588ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.008+0000 7f0588ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.008+0000 7f0588ff9640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.008+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0578004eb0 con 0x7f058c0b4330 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893725324 0 0) 0x7f058c144350 con 0x7f058c1470b0 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f058c141730 con 0x7f058c1470b0 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 10 client.?.objecter ms_handle_connect 0x7f058c1439b0 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 839831146 0 0) 0x7f0578004eb0 con 0x7f058c0b4330 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f058c144350 con 0x7f058c0b4330 2024-03-29T11:44:44.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929449324 0 0) 0x7f058c141460 con 0x7f058c1439b0 2024-03-29T11:44:44.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:44.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:44.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:44.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:44:44.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0578004eb0 con 0x7f058c1439b0 2024-03-29T11:44:44.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f057c001b00 con 0x7f058c0b4330 2024-03-29T11:44:44.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:44.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:44.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.009+0000 7f0588ff9640 10 monclient(hunting): dump: 2024-03-29T11:44:44.022 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:44.022 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:44.022 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:44.022 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:44.022 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 236437299 0 0) 0x7f058c141730 con 0x7f058c1470b0 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f058c141460 con 0x7f058c1470b0 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0584001dd0 con 0x7f058c1470b0 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 10 monclient(hunting): dump: 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:44.023 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2501983849 0 0) 0x7f0578004eb0 con 0x7f058c1439b0 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f058c141730 con 0x7f058c1439b0 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0580001670 con 0x7f058c1439b0 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 10 monclient(hunting): dump: 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:44.024 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3139723953 0 0) 0x7f058c141460 con 0x7f058c1470b0 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.010+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 >> v1:172.21.15.146:6789/0 conn(0x7f058c0b4330 legacy=0x7f058c120290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.011+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 >> v1:172.21.15.37:6789/0 conn(0x7f058c1439b0 legacy=0x7f058c140640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.011+0000 7f0588ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.011+0000 7f0588ff9640 1 monclient: found mon.c 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.011+0000 7f0588ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.011+0000 7f0588ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:44:44.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.011+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c11fd90 con 0x7f058c1470b0 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.011+0000 7f0588ff9640 10 monclient: _finish_auth 0 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.011+0000 7f0588ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.011+0000 7f0588ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.011+0000 7f059236b640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.011+0000 7f059236b640 10 monclient: _renew_subs 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.011+0000 7f059236b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.011+0000 7f059236b640 1 -- 172.21.15.37:0/157096349 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058c11f900 con 0x7f058c1470b0 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.011+0000 7f059236b640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.012+0000 7f059236b640 10 monclient: _renew_subs 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.012+0000 7f059236b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.012+0000 7f059236b640 1 -- 172.21.15.37:0/157096349 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f058c1dfe30 con 0x7f058c1470b0 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.012+0000 7f059236b640 1 librados: init done 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.014+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0584001f30 con 0x7f058c1470b0 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.014+0000 7f0588ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:44.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.014+0000 7f0589ffb640 4 set_mon_vals no callback set 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.014+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f0584003dc0 con 0x7f058c1470b0 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.014+0000 7f0588ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.014+0000 7f0588ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.014+0000 7f0588ff9640 10 monclient: dump: 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:44.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.014+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f0584003fe0 con 0x7f058c1470b0 2024-03-29T11:44:44.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.014+0000 7f05727fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:44.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.015+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f0584084d60 con 0x7f058c1470b0 2024-03-29T11:44:44.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.015+0000 7f0588ff9640 10 client.4631.objecter ms_dispatch 0x7f058c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:44.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.015+0000 7f0588ff9640 3 client.4631.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:44.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.016+0000 7f0588ff9640 3 client.4631.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:44.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.016+0000 7f0588ff9640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T11:44:44.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.016+0000 7f05727fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:44.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.016+0000 7f05727fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:44:44.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.016+0000 7f05727fc640 1 -- 172.21.15.37:0/157096349 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056c005080 con 0x7f058c1470b0 2024-03-29T11:44:44.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.016+0000 7f058bfff640 10 monclient: get_auth_request con 0x7f057806e0d0 auth_method 0 2024-03-29T11:44:44.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.017+0000 7f0588ff9640 10 client.4631.objecter ms_handle_connect 0x7f057806e0d0 2024-03-29T11:44:44.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.025+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f05840507c0 con 0x7f058c1470b0 2024-03-29T11:44:44.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.025+0000 7f0588ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:44.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.025+0000 7f0588ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:44.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.360+0000 7f05727fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:44.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.360+0000 7f05727fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:44.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.360+0000 7f05727fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:44:44.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.360+0000 7f05727fc640 1 -- 172.21.15.37:0/157096349 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f056c005330 con 0x7f058c1470b0 2024-03-29T11:44:44.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.362+0000 7f0588ff9640 1 -- 172.21.15.37:0/157096349 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f05840526f0 con 0x7f058c1470b0 2024-03-29T11:44:44.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.362+0000 7f0588ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:44.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.362+0000 7f0588ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:44.367 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:44:44.367 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:44.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.365+0000 7f05727fc640 10 librados: watch_flush enter 2024-03-29T11:44:44.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.365+0000 7f05727fc640 10 librados: watch_flush exit 2024-03-29T11:44:44.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.365+0000 7f05727fc640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T11:44:44.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.365+0000 7f05727fc640 10 client.4631.objecter successfully canceled tick 2024-03-29T11:44:44.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.368+0000 7f05727fc640 1 -- 172.21.15.37:0/157096349 >> v1:172.21.15.146:6816/2899149139 conn(0x7f057806e0d0 legacy=0x7f0578070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:44.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.368+0000 7f05727fc640 10 monclient: shutdown 2024-03-29T11:44:44.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.368+0000 7f05727fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:44.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.368+0000 7f05727fc640 1 -- 172.21.15.37:0/157096349 >> v1:172.21.15.37:6790/0 conn(0x7f058c1470b0 legacy=0x7f058c140d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:44.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.369+0000 7f05727fc640 1 -- 172.21.15.37:0/157096349 shutdown_connections 2024-03-29T11:44:44.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.369+0000 7f05727fc640 1 -- 172.21.15.37:0/157096349 >> 172.21.15.37:0/157096349 conn(0x7f058c066a00 msgr2=0x7f058c1d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:44.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.371+0000 7f05727fc640 1 -- 172.21.15.37:0/157096349 shutdown_connections 2024-03-29T11:44:44.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.372+0000 7f05727fc640 1 -- 172.21.15.37:0/157096349 wait complete. 2024-03-29T11:44:44.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:44.372+0000 7f05727fc640 1 librados: shutdown 2024-03-29T11:44:44.392 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:51.200 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:51.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.346+0000 7f9d261f4640 10 monclient: get_monmap_and_config 2024-03-29T11:44:51.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.346+0000 7f9d261f4640 10 monclient: build_initial_monmap 2024-03-29T11:44:51.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.346+0000 7f9d261f4640 10 monclient: monmap: 2024-03-29T11:44:51.353 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:51.353 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:51.353 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:44:51.347697+0000 2024-03-29T11:44:51.354 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:44:51.347697+0000 2024-03-29T11:44:51.354 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:51.354 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:51.354 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:51.354 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:44:51.354 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:44:51.354 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:51.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.347+0000 7f9d261f4640 1 Processor -- start 2024-03-29T11:44:51.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.348+0000 7f9d261f4640 1 -- start start 2024-03-29T11:44:51.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.348+0000 7f9d261f4640 10 monclient: init 2024-03-29T11:44:51.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.349+0000 7f9d261f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:51.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.349+0000 7f9d261f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:51.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.349+0000 7f9d261f4640 10 monclient(hunting): picked mon.noname-a con 0x7f9d200a96c0 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:51.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.349+0000 7f9d261f4640 10 monclient(hunting): picked mon.noname-c con 0x7f9d200a9aa0 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:51.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.349+0000 7f9d1ffff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f9d200ad730 0x7f9d2014dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:55124/0 (socket says 172.21.15.37:55124) 2024-03-29T11:44:51.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.349+0000 7f9d1ffff640 1 -- 172.21.15.37:0/3853181509 learned_addr learned my addr 172.21.15.37:0/3853181509 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:51.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.349+0000 7f9d261f4640 10 monclient(hunting): picked mon.noname-b con 0x7f9d200ad730 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:51.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.349+0000 7f9d261f4640 1 -- 172.21.15.37:0/3853181509 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2013fee0 con 0x7f9d200a96c0 2024-03-29T11:44:51.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.349+0000 7f9d261f4640 1 -- 172.21.15.37:0/3853181509 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d200aab80 con 0x7f9d200a9aa0 2024-03-29T11:44:51.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.350+0000 7f9d261f4640 1 -- 172.21.15.37:0/3853181509 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d20145e80 con 0x7f9d200ad730 2024-03-29T11:44:51.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.350+0000 7f9d261f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:51.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.350+0000 7f9d261f4640 10 monclient(hunting): authenticate will time out at 1470.799201s 2024-03-29T11:44:51.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.350+0000 7f9d1e7fc640 1 -- 172.21.15.37:0/3853181509 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910774989 0 0) 0x7f9d20145e80 con 0x7f9d200ad730 2024-03-29T11:44:51.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.351+0000 7f9d1e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:51.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.351+0000 7f9d1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:51.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.351+0000 7f9d1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:51.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.351+0000 7f9d1e7fc640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:44:51.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.351+0000 7f9d1e7fc640 1 -- 172.21.15.37:0/3853181509 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d08003520 con 0x7f9d200ad730 2024-03-29T11:44:51.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.351+0000 7f9d1e7fc640 1 -- 172.21.15.37:0/3853181509 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1895367482 0 0) 0x7f9d08003520 con 0x7f9d200ad730 2024-03-29T11:44:51.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.351+0000 7f9d1e7fc640 1 -- 172.21.15.37:0/3853181509 >> v1:172.21.15.146:6789/0 conn(0x7f9d200a9aa0 legacy=0x7f9d20145770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:51.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 1 -- 172.21.15.37:0/3853181509 >> v1:172.21.15.37:6789/0 conn(0x7f9d200a96c0 legacy=0x7f9d2013f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:51.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:51.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:44:51.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:51.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:44:51.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 1 -- 172.21.15.37:0/3853181509 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d200b4440 con 0x7f9d200ad730 2024-03-29T11:44:51.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:51.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:51.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:51.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 1 -- 172.21.15.37:0/3853181509 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9d10001690 con 0x7f9d200ad730 2024-03-29T11:44:51.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:51.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:51.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 10 monclient: dump: 2024-03-29T11:44:51.362 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:51.362 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:51.362 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:51.362 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:51.363 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:51.363 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:51.363 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:51.363 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:51.363 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:51.363 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:51.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d261f4640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:44:51.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d261f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:51.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 1 -- 172.21.15.37:0/3853181509 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d10001860 con 0x7f9d200ad730 2024-03-29T11:44:51.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:51.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 1 -- 172.21.15.37:0/3853181509 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9d10001cb0 con 0x7f9d200ad730 2024-03-29T11:44:51.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:51.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:51.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d1e7fc640 10 monclient: dump: 2024-03-29T11:44:51.365 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:51.365 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:51.365 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:51.365 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:51.365 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:51.365 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:51.365 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:51.366 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:51.366 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:51.366 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:51.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d261f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:51.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.352+0000 7f9d261f4640 4 set_mon_vals no callback set 2024-03-29T11:44:51.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.353+0000 7f9d261f4640 10 monclient: shutdown 2024-03-29T11:44:51.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.353+0000 7f9d261f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:51.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.353+0000 7f9d261f4640 1 -- 172.21.15.37:0/3853181509 >> v1:172.21.15.37:6790/0 conn(0x7f9d200ad730 legacy=0x7f9d2014dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:51.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.353+0000 7f9d261f4640 1 -- 172.21.15.37:0/3853181509 shutdown_connections 2024-03-29T11:44:51.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.353+0000 7f9d261f4640 1 -- 172.21.15.37:0/3853181509 >> 172.21.15.37:0/3853181509 conn(0x7f9d200b35e0 msgr2=0x7f9d200b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:51.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.353+0000 7f9d261f4640 1 -- 172.21.15.37:0/3853181509 shutdown_connections 2024-03-29T11:44:51.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.353+0000 7f9d261f4640 1 -- 172.21.15.37:0/3853181509 wait complete. 2024-03-29T11:44:51.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.354+0000 7f9d261f4640 10 monclient: build_initial_monmap 2024-03-29T11:44:51.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.354+0000 7f9d261f4640 10 monclient: monmap: 2024-03-29T11:44:51.367 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:51.367 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:51.367 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:44:51.367 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:44:51.367 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:51.368 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:51.368 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:51.368 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:44:51.368 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:44:51.368 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:51.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.357+0000 7f9d261f4640 1 librados: starting msgr at 2024-03-29T11:44:51.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.357+0000 7f9d261f4640 1 librados: starting objecter 2024-03-29T11:44:51.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.357+0000 7f9d261f4640 1 Processor -- start 2024-03-29T11:44:51.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.358+0000 7f9d261f4640 1 -- start start 2024-03-29T11:44:51.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.358+0000 7f9d261f4640 1 librados: setting wanted keys 2024-03-29T11:44:51.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.358+0000 7f9d261f4640 1 librados: calling monclient init 2024-03-29T11:44:51.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.358+0000 7f9d261f4640 10 monclient: init 2024-03-29T11:44:51.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.359+0000 7f9d261f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:51.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.359+0000 7f9d261f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:51.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.359+0000 7f9d261f4640 10 monclient(hunting): picked mon.noname-b con 0x7f9d200a96c0 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:51.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.359+0000 7f9d261f4640 10 monclient(hunting): picked mon.noname-c con 0x7f9d200ad730 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:51.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.359+0000 7f9d1effd640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f9d200ad730 0x7f9d20141260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:59352/0 (socket says 172.21.15.37:59352) 2024-03-29T11:44:51.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.363+0000 7f9d1effd640 1 -- 172.21.15.37:0/2732094795 learned_addr learned my addr 172.21.15.37:0/2732094795 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:51.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.363+0000 7f9d07fff640 10 client.?.objecter ms_handle_connect 0x7f9d200ad730 2024-03-29T11:44:51.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.363+0000 7f9d07fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:51.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.363+0000 7f9d261f4640 10 monclient(hunting): picked mon.noname-a con 0x7f9d2013d0a0 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:51.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.363+0000 7f9d261f4640 1 -- 172.21.15.37:0/2732094795 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2013dae0 con 0x7f9d2013d0a0 2024-03-29T11:44:51.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.363+0000 7f9d07fff640 10 client.?.objecter ms_handle_connect 0x7f9d2013d0a0 2024-03-29T11:44:51.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.364+0000 7f9d07fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:51.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.364+0000 7f9d261f4640 1 -- 172.21.15.37:0/2732094795 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2013e6a0 con 0x7f9d200a96c0 2024-03-29T11:44:51.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.364+0000 7f9d261f4640 1 -- 172.21.15.37:0/2732094795 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2013e980 con 0x7f9d200ad730 2024-03-29T11:44:51.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.364+0000 7f9d261f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:51.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.364+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919674882 0 0) 0x7f9d2013dae0 con 0x7f9d2013d0a0 2024-03-29T11:44:51.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.364+0000 7f9d261f4640 10 monclient(hunting): authenticate will time out at 1470.813416s 2024-03-29T11:44:51.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.364+0000 7f9d07fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:51.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.364+0000 7f9d07fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:51.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.364+0000 7f9d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:51.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.364+0000 7f9d07fff640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:44:51.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.364+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d0c004eb0 con 0x7f9d2013d0a0 2024-03-29T11:44:51.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.364+0000 7f9d07fff640 10 client.?.objecter ms_handle_connect 0x7f9d200a96c0 2024-03-29T11:44:51.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.364+0000 7f9d07fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:51.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.365+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399386371 0 0) 0x7f9d2013e980 con 0x7f9d200ad730 2024-03-29T11:44:51.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.365+0000 7f9d07fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:51.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.365+0000 7f9d07fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:51.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.365+0000 7f9d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:51.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.365+0000 7f9d07fff640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:44:51.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.365+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d2013dae0 con 0x7f9d200ad730 2024-03-29T11:44:51.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.365+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395830682 0 0) 0x7f9d2013e6a0 con 0x7f9d200a96c0 2024-03-29T11:44:51.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.365+0000 7f9d07fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:51.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.365+0000 7f9d07fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:51.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.365+0000 7f9d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:51.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.365+0000 7f9d07fff640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:44:51.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.365+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d2013e980 con 0x7f9d200a96c0 2024-03-29T11:44:51.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.365+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1403370716 0 0) 0x7f9d0c004eb0 con 0x7f9d2013d0a0 2024-03-29T11:44:51.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.365+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d2013e6a0 con 0x7f9d2013d0a0 2024-03-29T11:44:51.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1001940207 0 0) 0x7f9d2013dae0 con 0x7f9d200ad730 2024-03-29T11:44:51.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d0c004eb0 con 0x7f9d200ad730 2024-03-29T11:44:51.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9d10001b70 con 0x7f9d2013d0a0 2024-03-29T11:44:51.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:51.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:51.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 10 monclient(hunting): dump: 2024-03-29T11:44:51.391 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:51.391 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:51.391 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:51.391 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:51.392 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:51.392 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:51.392 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:51.392 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:51.393 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:51.393 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:51.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:51.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9d14001770 con 0x7f9d200ad730 2024-03-29T11:44:51.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:51.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:51.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 10 monclient(hunting): dump: 2024-03-29T11:44:51.394 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:51.394 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:51.394 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:51.395 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:51.395 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:51.395 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:51.395 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:51.395 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:51.395 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:51.395 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:51.396 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3220198125 0 0) 0x7f9d2013e6a0 con 0x7f9d2013d0a0 2024-03-29T11:44:51.396 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 >> v1:172.21.15.37:6790/0 conn(0x7f9d200ad730 legacy=0x7f9d20141260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:51.396 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 >> v1:172.21.15.146:6789/0 conn(0x7f9d200a96c0 legacy=0x7f9d2013c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:51.396 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.366+0000 7f9d07fff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:51.396 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d07fff640 1 monclient: found mon.a 2024-03-29T11:44:51.396 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:51.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d07fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:51.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2013d5c0 con 0x7f9d2013d0a0 2024-03-29T11:44:51.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d07fff640 10 monclient: _finish_auth 0 2024-03-29T11:44:51.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d07fff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:51.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:51.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d10001cd0 con 0x7f9d2013d0a0 2024-03-29T11:44:51.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:51.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f9d10004d90 con 0x7f9d2013d0a0 2024-03-29T11:44:51.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:51.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d1dffb640 4 set_mon_vals no callback set 2024-03-29T11:44:51.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d07fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:51.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d07fff640 10 monclient: dump: 2024-03-29T11:44:51.401 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:51.401 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:51.401 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:51.401 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:51.401 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:51.402 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:51.402 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:51.402 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:51.402 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:51.402 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:51.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d261f4640 5 monclient: authenticate success, global_id 4566 2024-03-29T11:44:51.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d261f4640 10 monclient: _renew_subs 2024-03-29T11:44:51.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.367+0000 7f9d261f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:51.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.368+0000 7f9d261f4640 1 -- 172.21.15.37:0/2732094795 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d20141970 con 0x7f9d2013d0a0 2024-03-29T11:44:51.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.368+0000 7f9d261f4640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:51.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.368+0000 7f9d261f4640 10 monclient: _renew_subs 2024-03-29T11:44:51.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.368+0000 7f9d261f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:51.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.368+0000 7f9d261f4640 1 -- 172.21.15.37:0/2732094795 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d201383b0 con 0x7f9d2013d0a0 2024-03-29T11:44:51.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.368+0000 7f9d261f4640 1 librados: init done 2024-03-29T11:44:51.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.370+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f9d1001ad60 con 0x7f9d2013d0a0 2024-03-29T11:44:51.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.370+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f9d10084a30 con 0x7f9d2013d0a0 2024-03-29T11:44:51.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.371+0000 7f9d07fff640 10 client.4566.objecter ms_dispatch 0x7f9d20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:51.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.371+0000 7f9d07fff640 3 client.4566.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:51.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.371+0000 7f9d07fff640 3 client.4566.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:51.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.371+0000 7f9d07fff640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-29T11:44:51.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.372+0000 7f9d261f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:51.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.372+0000 7f9d261f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:51.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.372+0000 7f9d261f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:51.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.372+0000 7f9d261f4640 1 -- 172.21.15.37:0/2732094795 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d20075f10 con 0x7f9d2013d0a0 2024-03-29T11:44:51.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.373+0000 7f9d1f7fe640 10 monclient: get_auth_request con 0x7f9d0c06e030 auth_method 0 2024-03-29T11:44:51.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.376+0000 7f9d07fff640 10 client.4566.objecter ms_handle_connect 0x7f9d0c06e030 2024-03-29T11:44:51.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.383+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9d10050310 con 0x7f9d2013d0a0 2024-03-29T11:44:51.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.383+0000 7f9d07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:51.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.383+0000 7f9d07fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:51.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.716+0000 7f9d261f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:51.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.716+0000 7f9d261f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:51.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.716+0000 7f9d261f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:44:51.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.716+0000 7f9d261f4640 1 -- 172.21.15.37:0/2732094795 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d200b1f40 con 0x7f9d2013d0a0 2024-03-29T11:44:51.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.717+0000 7f9d07fff640 1 -- 172.21.15.37:0/2732094795 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f9d10052240 con 0x7f9d2013d0a0 2024-03-29T11:44:51.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.717+0000 7f9d07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:51.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.717+0000 7f9d07fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:51.725 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:44:51.725 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:51.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.720+0000 7f9d261f4640 10 librados: watch_flush enter 2024-03-29T11:44:51.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.720+0000 7f9d261f4640 10 librados: watch_flush exit 2024-03-29T11:44:51.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.720+0000 7f9d261f4640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-29T11:44:51.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.720+0000 7f9d261f4640 10 client.4566.objecter successfully canceled tick 2024-03-29T11:44:51.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.721+0000 7f9d261f4640 1 -- 172.21.15.37:0/2732094795 >> v1:172.21.15.146:6816/2899149139 conn(0x7f9d0c06e030 legacy=0x7f9d0c0704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:51.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.721+0000 7f9d261f4640 10 monclient: shutdown 2024-03-29T11:44:51.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.721+0000 7f9d261f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:51.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.721+0000 7f9d261f4640 1 -- 172.21.15.37:0/2732094795 >> v1:172.21.15.37:6789/0 conn(0x7f9d2013d0a0 legacy=0x7f9d20137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:51.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.722+0000 7f9d261f4640 1 -- 172.21.15.37:0/2732094795 shutdown_connections 2024-03-29T11:44:51.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.722+0000 7f9d261f4640 1 -- 172.21.15.37:0/2732094795 >> 172.21.15.37:0/2732094795 conn(0x7f9d200b35e0 msgr2=0x7f9d200b2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:51.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.722+0000 7f9d261f4640 1 -- 172.21.15.37:0/2732094795 shutdown_connections 2024-03-29T11:44:51.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.722+0000 7f9d261f4640 1 -- 172.21.15.37:0/2732094795 wait complete. 2024-03-29T11:44:51.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:51.722+0000 7f9d261f4640 1 librados: shutdown 2024-03-29T11:44:51.745 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:58.556 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:58.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.688+0000 7f4315c17640 10 monclient: get_monmap_and_config 2024-03-29T11:44:58.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.688+0000 7f4315c17640 10 monclient: build_initial_monmap 2024-03-29T11:44:58.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.688+0000 7f4315c17640 10 monclient: monmap: 2024-03-29T11:44:58.692 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:58.692 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:58.692 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:44:58.688986+0000 2024-03-29T11:44:58.692 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:44:58.688986+0000 2024-03-29T11:44:58.692 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:58.693 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:58.693 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:58.693 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:44:58.693 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:44:58.693 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:58.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.689+0000 7f4315c17640 1 Processor -- start 2024-03-29T11:44:58.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.689+0000 7f4315c17640 1 -- start start 2024-03-29T11:44:58.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.689+0000 7f4315c17640 10 monclient: init 2024-03-29T11:44:58.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.690+0000 7f4315c17640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:58.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.690+0000 7f4315c17640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:58.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.690+0000 7f4315c17640 10 monclient(hunting): picked mon.noname-b con 0x7f43101422b0 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:58.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.690+0000 7f430f7fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f43101422b0 0x7f431013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:34946/0 (socket says 172.21.15.37:34946) 2024-03-29T11:44:58.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.690+0000 7f430f7fe640 1 -- 172.21.15.37:0/169596635 learned_addr learned my addr 172.21.15.37:0/169596635 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:58.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.690+0000 7f4315c17640 10 monclient(hunting): picked mon.noname-a con 0x7f4310142690 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:58.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.690+0000 7f4315c17640 10 monclient(hunting): picked mon.noname-c con 0x7f43100b17d0 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:58.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.690+0000 7f4315c17640 1 -- 172.21.15.37:0/169596635 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4310142c80 con 0x7f4310142690 2024-03-29T11:44:58.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.690+0000 7f4315c17640 1 -- 172.21.15.37:0/169596635 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43100aae40 con 0x7f43100b17d0 2024-03-29T11:44:58.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.690+0000 7f4315c17640 1 -- 172.21.15.37:0/169596635 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431013fbd0 con 0x7f43101422b0 2024-03-29T11:44:58.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.690+0000 7f4315c17640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:58.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.690+0000 7f4315c17640 10 monclient(hunting): authenticate will time out at 1478.139660s 2024-03-29T11:44:58.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.690+0000 7f430e7fc640 1 -- 172.21.15.37:0/169596635 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445211750 0 0) 0x7f431013fbd0 con 0x7f43101422b0 2024-03-29T11:44:58.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.691+0000 7f430e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:58.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.691+0000 7f430e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:58.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.691+0000 7f430e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:58.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.691+0000 7f430e7fc640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:44:58.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.691+0000 7f430e7fc640 1 -- 172.21.15.37:0/169596635 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f8003520 con 0x7f43101422b0 2024-03-29T11:44:58.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.691+0000 7f430e7fc640 1 -- 172.21.15.37:0/169596635 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 318264815 0 0) 0x7f42f8003520 con 0x7f43101422b0 2024-03-29T11:44:58.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.691+0000 7f430e7fc640 1 -- 172.21.15.37:0/169596635 >> v1:172.21.15.146:6789/0 conn(0x7f43100b17d0 legacy=0x7f43100b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:58.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.691+0000 7f430e7fc640 1 -- 172.21.15.37:0/169596635 >> v1:172.21.15.37:6789/0 conn(0x7f4310142690 legacy=0x7f43100b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:58.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.692+0000 7f430e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:58.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.692+0000 7f430e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:44:58.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.692+0000 7f430e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:58.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.692+0000 7f430e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:44:58.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.692+0000 7f430e7fc640 1 -- 172.21.15.37:0/169596635 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4310151940 con 0x7f43101422b0 2024-03-29T11:44:58.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.692+0000 7f430e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:58.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.692+0000 7f430e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:58.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.692+0000 7f430e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:58.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.692+0000 7f430e7fc640 1 -- 172.21.15.37:0/169596635 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4300001690 con 0x7f43101422b0 2024-03-29T11:44:58.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.692+0000 7f430e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:58.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.692+0000 7f430e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:58.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.692+0000 7f430e7fc640 10 monclient: dump: 2024-03-29T11:44:58.699 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:58.699 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:58.699 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:58.699 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:58.699 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:58.699 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:58.699 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.692+0000 7f4315c17640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.692+0000 7f4315c17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.693+0000 7f430e7fc640 1 -- 172.21.15.37:0/169596635 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43000017f0 con 0x7f43101422b0 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.693+0000 7f430e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.693+0000 7f430e7fc640 1 -- 172.21.15.37:0/169596635 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4300001c40 con 0x7f43101422b0 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.693+0000 7f430e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.693+0000 7f430e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.693+0000 7f430e7fc640 10 monclient: dump: 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:58.700 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.693+0000 7f4315c17640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.693+0000 7f4315c17640 4 set_mon_vals no callback set 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.693+0000 7f4315c17640 10 monclient: shutdown 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.694+0000 7f4315c17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.694+0000 7f4315c17640 1 -- 172.21.15.37:0/169596635 >> v1:172.21.15.37:6790/0 conn(0x7f43101422b0 legacy=0x7f431013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.694+0000 7f4315c17640 1 -- 172.21.15.37:0/169596635 shutdown_connections 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.694+0000 7f4315c17640 1 -- 172.21.15.37:0/169596635 >> 172.21.15.37:0/169596635 conn(0x7f4310066a00 msgr2=0x7f4310066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.694+0000 7f4315c17640 1 -- 172.21.15.37:0/169596635 shutdown_connections 2024-03-29T11:44:58.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.694+0000 7f4315c17640 1 -- 172.21.15.37:0/169596635 wait complete. 2024-03-29T11:44:58.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.695+0000 7f4315c17640 10 monclient: build_initial_monmap 2024-03-29T11:44:58.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.695+0000 7f4315c17640 10 monclient: monmap: 2024-03-29T11:44:58.706 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:44:58.706 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:58.706 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:44:58.707 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:44:58.707 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:58.707 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:58.707 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:44:58.707 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:44:58.707 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:44:58.707 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:58.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.695+0000 7f4315c17640 1 librados: starting msgr at 2024-03-29T11:44:58.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.695+0000 7f4315c17640 1 librados: starting objecter 2024-03-29T11:44:58.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.695+0000 7f4315c17640 1 Processor -- start 2024-03-29T11:44:58.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.696+0000 7f4315c17640 1 -- start start 2024-03-29T11:44:58.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.696+0000 7f4315c17640 1 librados: setting wanted keys 2024-03-29T11:44:58.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.696+0000 7f4315c17640 1 librados: calling monclient init 2024-03-29T11:44:58.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.696+0000 7f4315c17640 10 monclient: init 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.697+0000 7f4315c17640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.697+0000 7f4315c17640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.697+0000 7f4315c17640 10 monclient(hunting): picked mon.noname-b con 0x7f43100b17d0 addr v1:172.21.15.146:6789/0 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.698+0000 7f4315c17640 10 monclient(hunting): picked mon.noname-a con 0x7f43101422b0 addr v1:172.21.15.37:6789/0 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.698+0000 7f4315c17640 10 monclient(hunting): picked mon.noname-c con 0x7f4310142690 addr v1:172.21.15.37:6790/0 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.698+0000 7f4315c17640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43100ac4f0 con 0x7f43101422b0 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.698+0000 7f4315c17640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43100acd20 con 0x7f43100b17d0 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.698+0000 7f4315c17640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43100ac760 con 0x7f4310142690 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.698+0000 7f4315c17640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.698+0000 7f430f7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f43100b17d0 0x7f431014d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:42466/0 (socket says 172.21.15.37:42466) 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.698+0000 7f4315c17640 10 monclient(hunting): authenticate will time out at 1478.147699s 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.698+0000 7f430f7fe640 1 -- 172.21.15.37:0/2597379894 learned_addr learned my addr 172.21.15.37:0/2597379894 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.698+0000 7f42f7fff640 10 client.?.objecter ms_handle_connect 0x7f4310142690 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.698+0000 7f42f7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:58.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115916735 0 0) 0x7f43100ac760 con 0x7f4310142690 2024-03-29T11:44:58.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:58.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:58.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:58.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:44:58.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42fc004eb0 con 0x7f4310142690 2024-03-29T11:44:58.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3605906006 0 0) 0x7f42fc004eb0 con 0x7f4310142690 2024-03-29T11:44:58.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43100ac760 con 0x7f4310142690 2024-03-29T11:44:58.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4308001c30 con 0x7f4310142690 2024-03-29T11:44:58.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:58.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:58.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 10 monclient(hunting): dump: 2024-03-29T11:44:58.709 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 265999593 0 0) 0x7f43100ac760 con 0x7f4310142690 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 >> v1:172.21.15.146:6789/0 conn(0x7f43100b17d0 legacy=0x7f431014d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.699+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 >> v1:172.21.15.37:6789/0 conn(0x7f43101422b0 legacy=0x7f431012fd30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.700+0000 7f42f7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:58.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.700+0000 7f42f7fff640 1 monclient: found mon.c 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.700+0000 7f42f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.700+0000 7f42f7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.700+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43100aca40 con 0x7f4310142690 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.700+0000 7f42f7fff640 10 monclient: _finish_auth 0 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.700+0000 7f42f7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.700+0000 7f42f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.700+0000 7f4315c17640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.700+0000 7f4315c17640 10 monclient: _renew_subs 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.700+0000 7f4315c17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.700+0000 7f4315c17640 1 -- 172.21.15.37:0/2597379894 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4310132570 con 0x7f4310142690 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.701+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4308001d90 con 0x7f4310142690 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.701+0000 7f42f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.701+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f4308003bf0 con 0x7f4310142690 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.701+0000 7f42f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:58.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.701+0000 7f42f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.701+0000 7f42f7fff640 10 monclient: dump: 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.701+0000 7f430dffb640 4 set_mon_vals no callback set 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.701+0000 7f4315c17640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.701+0000 7f4315c17640 10 monclient: _renew_subs 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.701+0000 7f4315c17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:44:58.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.701+0000 7f4315c17640 1 -- 172.21.15.37:0/2597379894 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f431014e2a0 con 0x7f4310142690 2024-03-29T11:44:58.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.701+0000 7f4315c17640 1 librados: init done 2024-03-29T11:44:58.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.703+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f4308003d50 con 0x7f4310142690 2024-03-29T11:44:58.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.703+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 76946503 0 0) 0x7f4308083c90 con 0x7f4310142690 2024-03-29T11:44:58.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.703+0000 7f42f7fff640 10 client.4643.objecter ms_dispatch 0x7f4310000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:58.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.703+0000 7f42f7fff640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:58.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.703+0000 7f42f7fff640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:58.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.703+0000 7f42f7fff640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T11:44:58.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.704+0000 7f430f7fe640 10 monclient: get_auth_request con 0x7f42fc06dc90 auth_method 0 2024-03-29T11:44:58.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.704+0000 7f42f7fff640 10 client.4643.objecter ms_handle_connect 0x7f42fc06dc90 2024-03-29T11:44:58.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.706+0000 7f4315c17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:58.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.706+0000 7f4315c17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:58.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.706+0000 7f4315c17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:44:58.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.706+0000 7f4315c17640 1 -- 172.21.15.37:0/2597379894 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4310075f10 con 0x7f4310142690 2024-03-29T11:44:58.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.712+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f430804f3a0 con 0x7f4310142690 2024-03-29T11:44:58.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.712+0000 7f42f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:58.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:58.712+0000 7f42f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:59.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.020+0000 7f4315c17640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:59.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.020+0000 7f4315c17640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:59.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.020+0000 7f4315c17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-29T11:44:59.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.020+0000 7f4315c17640 1 -- 172.21.15.37:0/2597379894 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43100b12d0 con 0x7f4310142690 2024-03-29T11:44:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.021+0000 7f42f7fff640 1 -- 172.21.15.37:0/2597379894 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3005974420) 0x7f43080514a0 con 0x7f4310142690 2024-03-29T11:44:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.021+0000 7f42f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:59.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.021+0000 7f42f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:59.025 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:44:59.025 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:35:01.651950+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:01.651950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:59.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.023+0000 7f4315c17640 10 librados: watch_flush enter 2024-03-29T11:44:59.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.023+0000 7f4315c17640 10 librados: watch_flush exit 2024-03-29T11:44:59.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.023+0000 7f4315c17640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T11:44:59.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.023+0000 7f4315c17640 10 client.4643.objecter successfully canceled tick 2024-03-29T11:44:59.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.024+0000 7f4315c17640 1 -- 172.21.15.37:0/2597379894 >> v1:172.21.15.146:6816/2899149139 conn(0x7f42fc06dc90 legacy=0x7f42fc070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:59.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.024+0000 7f4315c17640 10 monclient: shutdown 2024-03-29T11:44:59.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.024+0000 7f4315c17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:59.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.024+0000 7f4315c17640 1 -- 172.21.15.37:0/2597379894 >> v1:172.21.15.37:6790/0 conn(0x7f4310142690 legacy=0x7f4310130440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:59.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.024+0000 7f4315c17640 1 -- 172.21.15.37:0/2597379894 shutdown_connections 2024-03-29T11:44:59.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.024+0000 7f4315c17640 1 -- 172.21.15.37:0/2597379894 >> 172.21.15.37:0/2597379894 conn(0x7f4310066a00 msgr2=0x7f431013dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:59.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.024+0000 7f4315c17640 1 -- 172.21.15.37:0/2597379894 shutdown_connections 2024-03-29T11:44:59.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.024+0000 7f4315c17640 1 -- 172.21.15.37:0/2597379894 wait complete. 2024-03-29T11:44:59.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:44:59.024+0000 7f4315c17640 1 librados: shutdown 2024-03-29T11:44:59.041 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:05.848 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:05.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.991+0000 7f8fedfdc640 10 monclient: get_monmap_and_config 2024-03-29T11:45:05.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.992+0000 7f8fedfdc640 10 monclient: build_initial_monmap 2024-03-29T11:45:05.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.992+0000 7f8fedfdc640 10 monclient: monmap: 2024-03-29T11:45:05.996 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:05.996 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:05.996 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:45:05.992902+0000 2024-03-29T11:45:05.997 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:45:05.992902+0000 2024-03-29T11:45:05.997 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:05.997 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:05.997 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:05.997 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:45:05.997 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:45:05.997 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:05.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.993+0000 7f8fedfdc640 1 Processor -- start 2024-03-29T11:45:05.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.993+0000 7f8fedfdc640 1 -- start start 2024-03-29T11:45:05.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.993+0000 7f8fedfdc640 10 monclient: init 2024-03-29T11:45:05.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.994+0000 7f8fedfdc640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:05.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.994+0000 7f8fedfdc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:06.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.996+0000 7f8fecfda640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8fe0006cb0 0x7f8fe00323e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47935/0 (socket says 172.21.15.37:47935) 2024-03-29T11:45:06.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.996+0000 7f8fedfdc640 10 monclient(hunting): picked mon.noname-a con 0x7f8fe0006cb0 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:06.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.996+0000 7f8fedfdc640 10 monclient(hunting): picked mon.noname-b con 0x7f8fe00304b0 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:06.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.996+0000 7f8fedfdc640 10 monclient(hunting): picked mon.noname-c con 0x7f8fe00cbcd0 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:06.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.996+0000 7f8fedfdc640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe00339e0 con 0x7f8fe0006cb0 2024-03-29T11:45:06.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.996+0000 7f8fedfdc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe00cc120 con 0x7f8fe00cbcd0 2024-03-29T11:45:06.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.996+0000 7f8fedfdc640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe0037120 con 0x7f8fe00304b0 2024-03-29T11:45:06.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.996+0000 7f8fedfdc640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:06.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.996+0000 7f8fedfdc640 10 monclient(hunting): authenticate will time out at 1485.445793s 2024-03-29T11:45:06.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:05.996+0000 7f8fecfda640 1 -- 172.21.15.37:0/2286425457 learned_addr learned my addr 172.21.15.37:0/2286425457 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 1 -- 172.21.15.37:0/2286425457 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457833525 0 0) 0x7f8fe00339e0 con 0x7f8fe0006cb0 2024-03-29T11:45:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:45:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 1 -- 172.21.15.37:0/2286425457 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd8003520 con 0x7f8fe0006cb0 2024-03-29T11:45:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 1 -- 172.21.15.37:0/2286425457 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858630972 0 0) 0x7f8fe0037120 con 0x7f8fe00304b0 2024-03-29T11:45:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:45:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 1 -- 172.21.15.37:0/2286425457 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe00339e0 con 0x7f8fe00304b0 2024-03-29T11:45:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 1 -- 172.21.15.37:0/2286425457 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1546936984 0 0) 0x7f8fd8003520 con 0x7f8fe0006cb0 2024-03-29T11:45:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 1 -- 172.21.15.37:0/2286425457 >> v1:172.21.15.37:6790/0 conn(0x7f8fe00304b0 legacy=0x7f8fe0036a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 1 -- 172.21.15.37:0/2286425457 >> v1:172.21.15.146:6789/0 conn(0x7f8fe00cbcd0 legacy=0x7f8fe00d7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:45:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 1 monclient: found mon.noname-a 2024-03-29T11:45:06.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:06.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:45:06.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 1 -- 172.21.15.37:0/2286425457 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe0033c50 con 0x7f8fe0006cb0 2024-03-29T11:45:06.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient: _finish_auth 0 2024-03-29T11:45:06.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:45:06.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:06.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 1 -- 172.21.15.37:0/2286425457 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8fe804f730 con 0x7f8fe0006cb0 2024-03-29T11:45:06.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:06.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:06.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fe77fe640 10 monclient: dump: 2024-03-29T11:45:06.006 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:06.007 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:06.007 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:06.007 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:06.007 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:06.007 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:06.007 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:06.007 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:06.007 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:06.007 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:06.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fedfdc640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:45:06.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.000+0000 7f8fedfdc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:06.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.004+0000 7f8fe77fe640 1 -- 172.21.15.37:0/2286425457 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fe80008d0 con 0x7f8fe0006cb0 2024-03-29T11:45:06.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.004+0000 7f8fe77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:06.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.004+0000 7f8fe77fe640 1 -- 172.21.15.37:0/2286425457 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8fe80758b0 con 0x7f8fe0006cb0 2024-03-29T11:45:06.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.004+0000 7f8fe77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:06.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.004+0000 7f8fe77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:06.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.004+0000 7f8fe77fe640 10 monclient: dump: 2024-03-29T11:45:06.009 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:06.009 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:06.009 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:06.009 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:06.009 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:06.009 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:06.009 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:06.009 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:06.009 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:06.009 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:06.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.004+0000 7f8fedfdc640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:06.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.004+0000 7f8fedfdc640 4 set_mon_vals no callback set 2024-03-29T11:45:06.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.005+0000 7f8fedfdc640 10 monclient: shutdown 2024-03-29T11:45:06.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.005+0000 7f8fedfdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:06.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.005+0000 7f8fedfdc640 1 -- 172.21.15.37:0/2286425457 >> v1:172.21.15.37:6789/0 conn(0x7f8fe0006cb0 legacy=0x7f8fe00323e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:06.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.005+0000 7f8fedfdc640 1 -- 172.21.15.37:0/2286425457 shutdown_connections 2024-03-29T11:45:06.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.005+0000 7f8fedfdc640 1 -- 172.21.15.37:0/2286425457 >> 172.21.15.37:0/2286425457 conn(0x7f8fe0005c40 msgr2=0x7f8fe0006050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:06.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.008+0000 7f8fedfdc640 1 -- 172.21.15.37:0/2286425457 shutdown_connections 2024-03-29T11:45:06.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.009+0000 7f8fedfdc640 1 -- 172.21.15.37:0/2286425457 wait complete. 2024-03-29T11:45:06.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.009+0000 7f8fedfdc640 10 monclient: build_initial_monmap 2024-03-29T11:45:06.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.009+0000 7f8fedfdc640 10 monclient: monmap: 2024-03-29T11:45:06.014 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:06.014 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:06.014 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:45:06.015 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:45:06.015 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:06.015 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:06.015 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:06.015 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:45:06.015 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:45:06.015 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:06.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.010+0000 7f8fedfdc640 1 librados: starting msgr at 2024-03-29T11:45:06.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.010+0000 7f8fedfdc640 1 librados: starting objecter 2024-03-29T11:45:06.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.010+0000 7f8fedfdc640 1 Processor -- start 2024-03-29T11:45:06.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.012+0000 7f8fedfdc640 1 -- start start 2024-03-29T11:45:06.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.012+0000 7f8fedfdc640 1 librados: setting wanted keys 2024-03-29T11:45:06.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.012+0000 7f8fedfdc640 1 librados: calling monclient init 2024-03-29T11:45:06.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.012+0000 7f8fedfdc640 10 monclient: init 2024-03-29T11:45:06.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.013+0000 7f8fedfdc640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:06.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.013+0000 7f8fedfdc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.013+0000 7f8fedfdc640 10 monclient(hunting): picked mon.noname-b con 0x7f8fe0006cb0 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.013+0000 7f8fedfdc640 10 monclient(hunting): picked mon.noname-a con 0x7f8fe00304b0 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.013+0000 7f8fedfdc640 10 monclient(hunting): picked mon.noname-c con 0x7f8fe00cbcd0 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.013+0000 7f8fedfdc640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe000a820 con 0x7f8fe00304b0 2024-03-29T11:45:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.013+0000 7f8fedfdc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe000ab00 con 0x7f8fe0006cb0 2024-03-29T11:45:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.013+0000 7f8fedfdc640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe00097f0 con 0x7f8fe00cbcd0 2024-03-29T11:45:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.013+0000 7f8fedfdc640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.013+0000 7f8fedfdc640 10 monclient(hunting): authenticate will time out at 1485.462278s 2024-03-29T11:45:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.013+0000 7f8fed7db640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f8fe00cbcd0 0x7f8fe016bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:34964/0 (socket says 172.21.15.37:34964) 2024-03-29T11:45:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.013+0000 7f8fed7db640 1 -- 172.21.15.37:0/3961237865 learned_addr learned my addr 172.21.15.37:0/3961237865 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.013+0000 7f8fe57fa640 10 client.?.objecter ms_handle_connect 0x7f8fe00cbcd0 2024-03-29T11:45:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.013+0000 7f8fe57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:06.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.014+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455906124 0 0) 0x7f8fe00097f0 con 0x7f8fe00cbcd0 2024-03-29T11:45:06.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.014+0000 7f8fe57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:06.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.014+0000 7f8fe57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:06.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.014+0000 7f8fe57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:06.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.014+0000 7f8fe57fa640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:45:06.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.014+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fcc004eb0 con 0x7f8fe00cbcd0 2024-03-29T11:45:06.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.015+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3628549362 0 0) 0x7f8fcc004eb0 con 0x7f8fe00cbcd0 2024-03-29T11:45:06.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.015+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fe00097f0 con 0x7f8fe00cbcd0 2024-03-29T11:45:06.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.015+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8fdc0018e0 con 0x7f8fe00cbcd0 2024-03-29T11:45:06.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.015+0000 7f8fe57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:06.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.015+0000 7f8fe57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:06.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.015+0000 7f8fe57fa640 10 monclient(hunting): dump: 2024-03-29T11:45:06.024 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:06.024 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:06.024 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:06.024 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:06.024 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:06.024 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:06.024 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:06.024 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:06.024 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:06.024 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:06.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.015+0000 7f8fe57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:06.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.017+0000 7f8fe57fa640 10 client.?.objecter ms_handle_connect 0x7f8fe0006cb0 2024-03-29T11:45:06.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.017+0000 7f8fe57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:06.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.018+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214131731 0 0) 0x7f8fe000ab00 con 0x7f8fe0006cb0 2024-03-29T11:45:06.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.018+0000 7f8fe57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:06.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.018+0000 7f8fe57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:06.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.018+0000 7f8fe57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:06.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.018+0000 7f8fe57fa640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:45:06.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.018+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fcc0056d0 con 0x7f8fe0006cb0 2024-03-29T11:45:06.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.018+0000 7f8fe57fa640 10 client.?.objecter ms_handle_connect 0x7f8fe00304b0 2024-03-29T11:45:06.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.018+0000 7f8fe57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:06.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653506390 0 0) 0x7f8fe000a820 con 0x7f8fe00304b0 2024-03-29T11:45:06.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:06.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:06.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:06.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:45:06.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe000ab00 con 0x7f8fe00304b0 2024-03-29T11:45:06.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 74869702 0 0) 0x7f8fcc0056d0 con 0x7f8fe0006cb0 2024-03-29T11:45:06.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fcc005cd0 con 0x7f8fe0006cb0 2024-03-29T11:45:06.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8fe804f400 con 0x7f8fe0006cb0 2024-03-29T11:45:06.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 10 monclient(hunting): dump: 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2388047793 0 0) 0x7f8fe000ab00 con 0x7f8fe00304b0 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fcc0056d0 con 0x7f8fe00304b0 2024-03-29T11:45:06.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8fd4001790 con 0x7f8fe00304b0 2024-03-29T11:45:06.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:06.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:06.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 10 monclient(hunting): dump: 2024-03-29T11:45:06.034 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:06.034 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:06.034 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:06.034 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:06.034 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:06.034 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:06.034 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 889523343 0 0) 0x7f8fcc005cd0 con 0x7f8fe0006cb0 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 >> v1:172.21.15.37:6790/0 conn(0x7f8fe00cbcd0 legacy=0x7f8fe016bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.019+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 >> v1:172.21.15.37:6789/0 conn(0x7f8fe00304b0 legacy=0x7f8fe015e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe57fa640 1 monclient: found mon.b 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe0009980 con 0x7f8fe0006cb0 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe57fa640 10 monclient: _finish_auth 0 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:06.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fedfdc640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:45:06.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fedfdc640 10 monclient: _renew_subs 2024-03-29T11:45:06.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fedfdc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:06.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fedfdc640 1 -- 172.21.15.37:0/3961237865 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe000d610 con 0x7f8fe0006cb0 2024-03-29T11:45:06.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fedfdc640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:06.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fedfdc640 10 monclient: _renew_subs 2024-03-29T11:45:06.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fedfdc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:06.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fedfdc640 1 -- 172.21.15.37:0/3961237865 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe00cfd20 con 0x7f8fe0006cb0 2024-03-29T11:45:06.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fedfdc640 1 librados: init done 2024-03-29T11:45:06.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fe804fdd0 con 0x7f8fe0006cb0 2024-03-29T11:45:06.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:06.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8fe8067bf0 con 0x7f8fe0006cb0 2024-03-29T11:45:06.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:06.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:06.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe57fa640 10 monclient: dump: 2024-03-29T11:45:06.037 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:06.037 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:06.037 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:06.037 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:06.037 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:06.037 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:06.037 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:06.037 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:06.037 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:06.037 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:06.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.020+0000 7f8fe6ffd640 4 set_mon_vals no callback set 2024-03-29T11:45:06.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.021+0000 7f8fedfdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.022+0000 7f8fedfdc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.022+0000 7f8fedfdc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.022+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f8fe8067d50 con 0x7f8fe0006cb0 2024-03-29T11:45:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.022+0000 7f8fedfdc640 1 -- 172.21.15.37:0/3961237865 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc8005080 con 0x7f8fe0006cb0 2024-03-29T11:45:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.023+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2864318804 0 0) 0x7f8fe80cf160 con 0x7f8fe0006cb0 2024-03-29T11:45:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.023+0000 7f8fe57fa640 10 client.4525.objecter ms_dispatch 0x7f8fe8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.023+0000 7f8fe57fa640 3 client.4525.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.023+0000 7f8fe57fa640 3 client.4525.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.023+0000 7f8fe57fa640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-29T11:45:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.024+0000 7f8fe7fff640 10 monclient: get_auth_request con 0x7f8fcc06e570 auth_method 0 2024-03-29T11:45:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.024+0000 7f8fe57fa640 10 client.4525.objecter ms_handle_connect 0x7f8fcc06e570 2024-03-29T11:45:06.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.031+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8fe80cb400 con 0x7f8fe0006cb0 2024-03-29T11:45:06.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.031+0000 7f8fe57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:06.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.031+0000 7f8fe57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:06.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.368+0000 7f8fedfdc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:06.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.368+0000 7f8fedfdc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:06.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.368+0000 7f8fedfdc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:06.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.368+0000 7f8fedfdc640 1 -- 172.21.15.37:0/3961237865 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fc8005330 con 0x7f8fe0006cb0 2024-03-29T11:45:06.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.369+0000 7f8fe57fa640 1 -- 172.21.15.37:0/3961237865 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1684400214) 0x7f8fe80cb5a0 con 0x7f8fe0006cb0 2024-03-29T11:45:06.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.369+0000 7f8fe57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:06.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.369+0000 7f8fe57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:06.375 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:45:06.375 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:45:04.373638+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:04.373638+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.371+0000 7f8fedfdc640 10 librados: watch_flush enter 2024-03-29T11:45:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.371+0000 7f8fedfdc640 10 librados: watch_flush exit 2024-03-29T11:45:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.371+0000 7f8fedfdc640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-29T11:45:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.371+0000 7f8fedfdc640 10 client.4525.objecter successfully canceled tick 2024-03-29T11:45:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.371+0000 7f8fedfdc640 1 -- 172.21.15.37:0/3961237865 >> v1:172.21.15.146:6816/2899149139 conn(0x7f8fcc06e570 legacy=0x7f8fcc070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.371+0000 7f8fedfdc640 10 monclient: shutdown 2024-03-29T11:45:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.371+0000 7f8fedfdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.371+0000 7f8fedfdc640 1 -- 172.21.15.37:0/3961237865 >> v1:172.21.15.146:6789/0 conn(0x7f8fe0006cb0 legacy=0x7f8fe015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.372+0000 7f8fedfdc640 1 -- 172.21.15.37:0/3961237865 shutdown_connections 2024-03-29T11:45:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.372+0000 7f8fedfdc640 1 -- 172.21.15.37:0/3961237865 >> 172.21.15.37:0/3961237865 conn(0x7f8fe0005c40 msgr2=0x7f8fe00378e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.373+0000 7f8fedfdc640 1 -- 172.21.15.37:0/3961237865 shutdown_connections 2024-03-29T11:45:06.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.373+0000 7f8fedfdc640 1 -- 172.21.15.37:0/3961237865 wait complete. 2024-03-29T11:45:06.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:06.373+0000 7f8fedfdc640 1 librados: shutdown 2024-03-29T11:45:06.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:13.201 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:13.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.359+0000 7fc13008d640 10 monclient: get_monmap_and_config 2024-03-29T11:45:13.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.359+0000 7fc13008d640 10 monclient: build_initial_monmap 2024-03-29T11:45:13.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.360+0000 7fc13008d640 10 monclient: monmap: 2024-03-29T11:45:13.364 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:13.365 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:13.365 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:45:13.360862+0000 2024-03-29T11:45:13.365 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:45:13.360862+0000 2024-03-29T11:45:13.365 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:13.365 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:13.365 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:13.365 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:45:13.365 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:45:13.365 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:13.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.362+0000 7fc13008d640 1 Processor -- start 2024-03-29T11:45:13.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.362+0000 7fc13008d640 1 -- start start 2024-03-29T11:45:13.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.363+0000 7fc13008d640 10 monclient: init 2024-03-29T11:45:13.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.363+0000 7fc13008d640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:13.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.363+0000 7fc13008d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:13.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.363+0000 7fc13008d640 10 monclient(hunting): picked mon.noname-a con 0x7fc1281457f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:13.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.363+0000 7fc13008d640 10 monclient(hunting): picked mon.noname-c con 0x7fc12813f650 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:13.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.364+0000 7fc13008d640 10 monclient(hunting): picked mon.noname-b con 0x7fc1280a9d70 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:13.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.364+0000 7fc13008d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1280aaef0 con 0x7fc1281457f0 2024-03-29T11:45:13.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.364+0000 7fc13008d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc128145bd0 con 0x7fc12813f650 2024-03-29T11:45:13.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.364+0000 7fc13008d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc128145eb0 con 0x7fc1280a9d70 2024-03-29T11:45:13.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.364+0000 7fc13008d640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:13.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.364+0000 7fc13008d640 10 monclient(hunting): authenticate will time out at 1492.812978s 2024-03-29T11:45:13.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.370+0000 7fc12de29640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc1281457f0 0x7fc12813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42287/0 (socket says 172.21.15.37:42287) 2024-03-29T11:45:13.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.371+0000 7fc12de29640 1 -- 172.21.15.37:0/2592864145 learned_addr learned my addr 172.21.15.37:0/2592864145 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:13.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.374+0000 7fc12ce27640 1 -- 172.21.15.37:0/2592864145 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456225119 0 0) 0x7fc1280aaef0 con 0x7fc1281457f0 2024-03-29T11:45:13.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.374+0000 7fc12ce27640 10 monclient(hunting): _negotiate 2024-03-29T11:45:13.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.374+0000 7fc12ce27640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:13.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.374+0000 7fc12ce27640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:13.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.374+0000 7fc12ce27640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:45:13.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.374+0000 7fc12ce27640 1 -- 172.21.15.37:0/2592864145 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc114003520 con 0x7fc1281457f0 2024-03-29T11:45:13.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.374+0000 7fc12ce27640 1 -- 172.21.15.37:0/2592864145 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802628531 0 0) 0x7fc128145bd0 con 0x7fc12813f650 2024-03-29T11:45:13.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.374+0000 7fc12ce27640 10 monclient(hunting): _negotiate 2024-03-29T11:45:13.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.374+0000 7fc12ce27640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:13.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.374+0000 7fc12ce27640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:13.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.374+0000 7fc12ce27640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:45:13.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.374+0000 7fc12ce27640 1 -- 172.21.15.37:0/2592864145 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1280aaef0 con 0x7fc12813f650 2024-03-29T11:45:13.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 1 -- 172.21.15.37:0/2592864145 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2042995312 0 0) 0x7fc114003520 con 0x7fc1281457f0 2024-03-29T11:45:13.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 1 -- 172.21.15.37:0/2592864145 >> v1:172.21.15.37:6790/0 conn(0x7fc1280a9d70 legacy=0x7fc12814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:13.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 1 -- 172.21.15.37:0/2592864145 >> v1:172.21.15.146:6789/0 conn(0x7fc12813f650 legacy=0x7fc1280b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:13.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 10 monclient: _finish_hunting 0 2024-03-29T11:45:13.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 1 monclient: found mon.noname-a 2024-03-29T11:45:13.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:13.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:45:13.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 1 -- 172.21.15.37:0/2592864145 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc128146340 con 0x7fc1281457f0 2024-03-29T11:45:13.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 10 monclient: _finish_auth 0 2024-03-29T11:45:13.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 10 monclient: _check_auth_tickets 2024-03-29T11:45:13.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:13.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 1 -- 172.21.15.37:0/2592864145 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc124001690 con 0x7fc1281457f0 2024-03-29T11:45:13.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:13.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:13.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 10 monclient: dump: 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 1 -- 172.21.15.37:0/2592864145 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1240017f0 con 0x7fc1281457f0 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 1 -- 172.21.15.37:0/2592864145 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc124001c00 con 0x7fc1281457f0 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:13.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.376+0000 7fc12ce27640 10 monclient: dump: 2024-03-29T11:45:13.385 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:13.385 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:13.385 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:13.385 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:13.385 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:13.385 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:13.385 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:13.385 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:13.385 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:13.385 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:13.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.378+0000 7fc13008d640 5 monclient: authenticate success, global_id 4575 2024-03-29T11:45:13.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.378+0000 7fc13008d640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:13.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.378+0000 7fc13008d640 4 set_mon_vals no callback set 2024-03-29T11:45:13.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.379+0000 7fc13008d640 10 monclient: shutdown 2024-03-29T11:45:13.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.379+0000 7fc13008d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:13.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.379+0000 7fc13008d640 1 -- 172.21.15.37:0/2592864145 >> v1:172.21.15.37:6789/0 conn(0x7fc1281457f0 legacy=0x7fc12813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:13.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.379+0000 7fc13008d640 1 -- 172.21.15.37:0/2592864145 shutdown_connections 2024-03-29T11:45:13.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.379+0000 7fc13008d640 1 -- 172.21.15.37:0/2592864145 >> 172.21.15.37:0/2592864145 conn(0x7fc128066a00 msgr2=0x7fc128066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:13.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.379+0000 7fc13008d640 1 -- 172.21.15.37:0/2592864145 shutdown_connections 2024-03-29T11:45:13.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.379+0000 7fc13008d640 1 -- 172.21.15.37:0/2592864145 wait complete. 2024-03-29T11:45:13.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.380+0000 7fc13008d640 10 monclient: build_initial_monmap 2024-03-29T11:45:13.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.380+0000 7fc13008d640 10 monclient: monmap: 2024-03-29T11:45:13.388 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:13.388 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:13.388 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:45:13.388 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:45:13.388 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:13.388 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:13.388 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:13.388 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.381+0000 7fc13008d640 1 librados: starting msgr at 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.381+0000 7fc13008d640 1 librados: starting objecter 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.381+0000 7fc13008d640 1 Processor -- start 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.381+0000 7fc13008d640 1 -- start start 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.381+0000 7fc13008d640 1 librados: setting wanted keys 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.381+0000 7fc13008d640 1 librados: calling monclient init 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.381+0000 7fc13008d640 10 monclient: init 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.382+0000 7fc13008d640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.382+0000 7fc13008d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.382+0000 7fc13008d640 10 monclient(hunting): picked mon.noname-a con 0x7fc1280a9d70 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.382+0000 7fc13008d640 10 monclient(hunting): picked mon.noname-c con 0x7fc12813f650 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.382+0000 7fc13008d640 10 monclient(hunting): picked mon.noname-b con 0x7fc1281457f0 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:13.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.382+0000 7fc13008d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc128142900 con 0x7fc1280a9d70 2024-03-29T11:45:13.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.382+0000 7fc13008d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1281e1ad0 con 0x7fc1281457f0 2024-03-29T11:45:13.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.382+0000 7fc13008d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc128142b90 con 0x7fc12813f650 2024-03-29T11:45:13.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.382+0000 7fc13008d640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:13.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.382+0000 7fc13008d640 10 monclient(hunting): authenticate will time out at 1492.831267s 2024-03-29T11:45:13.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.388+0000 7fc12e62a640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc1281457f0 0x7fc128071ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:48787/0 (socket says 172.21.15.37:48787) 2024-03-29T11:45:13.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.388+0000 7fc12e62a640 1 -- 172.21.15.37:0/1435062539 learned_addr learned my addr 172.21.15.37:0/1435062539 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:13.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.391+0000 7fc11e7fc640 10 client.?.objecter ms_handle_connect 0x7fc1281457f0 2024-03-29T11:45:13.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.391+0000 7fc11e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:13.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.391+0000 7fc11e7fc640 10 client.?.objecter ms_handle_connect 0x7fc12813f650 2024-03-29T11:45:13.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.391+0000 7fc11e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:13.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.391+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810137975 0 0) 0x7fc1281e1ad0 con 0x7fc1281457f0 2024-03-29T11:45:13.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.391+0000 7fc11e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:13.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.391+0000 7fc11e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:13.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.391+0000 7fc11e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:13.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.391+0000 7fc11e7fc640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:45:13.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.392+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc10c004eb0 con 0x7fc1281457f0 2024-03-29T11:45:13.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.392+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645645417 0 0) 0x7fc128142b90 con 0x7fc12813f650 2024-03-29T11:45:13.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.392+0000 7fc11e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:13.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.392+0000 7fc11e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:13.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.392+0000 7fc11e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:13.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.392+0000 7fc11e7fc640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:45:13.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.392+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1281e1ad0 con 0x7fc12813f650 2024-03-29T11:45:13.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.392+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3007778803 0 0) 0x7fc10c004eb0 con 0x7fc1281457f0 2024-03-29T11:45:13.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.392+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc128142b90 con 0x7fc1281457f0 2024-03-29T11:45:13.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.392+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc120001d30 con 0x7fc1281457f0 2024-03-29T11:45:13.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.392+0000 7fc11e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:13.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.393+0000 7fc11e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:13.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.393+0000 7fc11e7fc640 10 monclient(hunting): dump: 2024-03-29T11:45:13.400 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:13.401 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:13.401 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:13.401 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:13.401 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:13.401 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:13.401 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:13.401 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:13.401 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:13.401 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:13.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.393+0000 7fc11e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:13.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.393+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 359455369 0 0) 0x7fc128142b90 con 0x7fc1281457f0 2024-03-29T11:45:13.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.393+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 >> v1:172.21.15.37:6790/0 conn(0x7fc12813f650 legacy=0x7fc1281421f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:13.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.393+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 >> v1:172.21.15.37:6789/0 conn(0x7fc1280a9d70 legacy=0x7fc128139aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:13.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.393+0000 7fc11e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:13.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.393+0000 7fc11e7fc640 1 monclient: found mon.b 2024-03-29T11:45:13.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.393+0000 7fc11e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:13.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.393+0000 7fc11e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:13.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.393+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1280721e0 con 0x7fc1281457f0 2024-03-29T11:45:13.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.393+0000 7fc11e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:13.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.393+0000 7fc11e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:13.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.394+0000 7fc11e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:13.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.394+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc120003990 con 0x7fc1281457f0 2024-03-29T11:45:13.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.394+0000 7fc13008d640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:45:13.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.394+0000 7fc13008d640 10 monclient: _renew_subs 2024-03-29T11:45:13.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.394+0000 7fc13008d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:13.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.395+0000 7fc13008d640 1 -- 172.21.15.37:0/1435062539 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc128141ef0 con 0x7fc1281457f0 2024-03-29T11:45:13.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.395+0000 7fc11e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:13.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.395+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fc120003ca0 con 0x7fc1281457f0 2024-03-29T11:45:13.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.395+0000 7fc11e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:13.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.395+0000 7fc11e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:13.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.395+0000 7fc11e7fc640 10 monclient: dump: 2024-03-29T11:45:13.405 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:13.405 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:13.405 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:13.405 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:13.405 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:13.405 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:13.405 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:13.405 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:13.405 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:13.406 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:13.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.395+0000 7fc11ffff640 4 set_mon_vals no callback set 2024-03-29T11:45:13.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.395+0000 7fc13008d640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:13.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.395+0000 7fc13008d640 10 monclient: _renew_subs 2024-03-29T11:45:13.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.395+0000 7fc13008d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:13.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.395+0000 7fc13008d640 1 -- 172.21.15.37:0/1435062539 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1281e76d0 con 0x7fc1281457f0 2024-03-29T11:45:13.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.395+0000 7fc13008d640 1 librados: init done 2024-03-29T11:45:13.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.397+0000 7fc13008d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:13.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.397+0000 7fc13008d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:13.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.397+0000 7fc13008d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:13.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.397+0000 7fc13008d640 1 -- 172.21.15.37:0/1435062539 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc128075f10 con 0x7fc1281457f0 2024-03-29T11:45:13.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.399+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fc120003e00 con 0x7fc1281457f0 2024-03-29T11:45:13.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.400+0000 7fc12de29640 10 monclient: get_auth_request con 0x7fc10c06dd60 auth_method 0 2024-03-29T11:45:13.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.401+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2864318804 0 0) 0x7fc120084e60 con 0x7fc1281457f0 2024-03-29T11:45:13.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.401+0000 7fc11e7fc640 10 client.4531.objecter ms_dispatch 0x7fc128000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:13.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.401+0000 7fc11e7fc640 3 client.4531.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:13.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.401+0000 7fc11e7fc640 3 client.4531.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:13.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.401+0000 7fc11e7fc640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-29T11:45:13.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.401+0000 7fc11e7fc640 10 client.4531.objecter ms_handle_connect 0x7fc10c06dd60 2024-03-29T11:45:13.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.410+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc120050740 con 0x7fc1281457f0 2024-03-29T11:45:13.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.410+0000 7fc11e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:13.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.410+0000 7fc11e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:13.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.748+0000 7fc13008d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:13.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.748+0000 7fc13008d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:13.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.748+0000 7fc13008d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:13.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.748+0000 7fc13008d640 1 -- 172.21.15.37:0/1435062539 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1280b3680 con 0x7fc1281457f0 2024-03-29T11:45:13.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.752+0000 7fc11e7fc640 1 -- 172.21.15.37:0/1435062539 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1684400214) 0x7fc120052670 con 0x7fc1281457f0 2024-03-29T11:45:13.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.752+0000 7fc11e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:13.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.752+0000 7fc11e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:13.758 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:45:13.758 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:45:04.373638+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:04.373638+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:13.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.756+0000 7fc13008d640 10 librados: watch_flush enter 2024-03-29T11:45:13.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.756+0000 7fc13008d640 10 librados: watch_flush exit 2024-03-29T11:45:13.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.756+0000 7fc13008d640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-29T11:45:13.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.756+0000 7fc13008d640 10 client.4531.objecter successfully canceled tick 2024-03-29T11:45:13.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.757+0000 7fc13008d640 1 -- 172.21.15.37:0/1435062539 >> v1:172.21.15.146:6816/2899149139 conn(0x7fc10c06dd60 legacy=0x7fc10c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:13.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.757+0000 7fc13008d640 10 monclient: shutdown 2024-03-29T11:45:13.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.757+0000 7fc13008d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:13.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.757+0000 7fc13008d640 1 -- 172.21.15.37:0/1435062539 >> v1:172.21.15.146:6789/0 conn(0x7fc1281457f0 legacy=0x7fc128071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:13.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.757+0000 7fc13008d640 1 -- 172.21.15.37:0/1435062539 shutdown_connections 2024-03-29T11:45:13.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.757+0000 7fc13008d640 1 -- 172.21.15.37:0/1435062539 >> 172.21.15.37:0/1435062539 conn(0x7fc128066a00 msgr2=0x7fc1280b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:13.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.757+0000 7fc13008d640 1 -- 172.21.15.37:0/1435062539 shutdown_connections 2024-03-29T11:45:13.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.758+0000 7fc13008d640 1 -- 172.21.15.37:0/1435062539 wait complete. 2024-03-29T11:45:13.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:13.758+0000 7fc13008d640 1 librados: shutdown 2024-03-29T11:45:13.783 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:20.590 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:20.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.733+0000 7f8c57fff640 10 monclient: get_monmap_and_config 2024-03-29T11:45:20.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.733+0000 7f8c57fff640 10 monclient: build_initial_monmap 2024-03-29T11:45:20.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.733+0000 7f8c57fff640 10 monclient: monmap: 2024-03-29T11:45:20.738 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:20.738 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:20.738 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:45:20.734290+0000 2024-03-29T11:45:20.738 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:45:20.734290+0000 2024-03-29T11:45:20.738 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:20.738 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:20.738 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:20.738 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:45:20.738 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:45:20.738 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:20.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.734+0000 7f8c57fff640 1 Processor -- start 2024-03-29T11:45:20.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.734+0000 7f8c57fff640 1 -- start start 2024-03-29T11:45:20.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.734+0000 7f8c57fff640 10 monclient: init 2024-03-29T11:45:20.746 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.736+0000 7f8c57fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:20.746 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.736+0000 7f8c57fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.736+0000 7f8c57fff640 10 monclient(hunting): picked mon.noname-b con 0x7f8c50006eb0 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.736+0000 7f8c57fff640 10 monclient(hunting): picked mon.noname-a con 0x7f8c50038f10 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.736+0000 7f8c57fff640 10 monclient(hunting): picked mon.noname-c con 0x7f8c50036690 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.736+0000 7f8c57fff640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c50033c90 con 0x7f8c50038f10 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.736+0000 7f8c57fff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c500389f0 con 0x7f8c50036690 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.736+0000 7f8c57fff640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c50030890 con 0x7f8c50006eb0 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.736+0000 7f8c57fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.736+0000 7f8c57fff640 10 monclient(hunting): authenticate will time out at 1500.185680s 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.740+0000 7f8c56ffd640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f8c50006eb0 0x7f8c50031760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:43325/0 (socket says 172.21.15.37:43325) 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.740+0000 7f8c56ffd640 1 -- 172.21.15.37:0/4134879158 learned_addr learned my addr 172.21.15.37:0/4134879158 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.741+0000 7f8c55ffb640 1 -- 172.21.15.37:0/4134879158 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118732609 0 0) 0x7f8c50030890 con 0x7f8c50006eb0 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.741+0000 7f8c55ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.741+0000 7f8c55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.741+0000 7f8c55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:20.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.741+0000 7f8c55ffb640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:45:20.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.741+0000 7f8c55ffb640 1 -- 172.21.15.37:0/4134879158 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c40003520 con 0x7f8c50006eb0 2024-03-29T11:45:20.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.742+0000 7f8c55ffb640 1 -- 172.21.15.37:0/4134879158 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1837413845 0 0) 0x7f8c40003520 con 0x7f8c50006eb0 2024-03-29T11:45:20.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.742+0000 7f8c55ffb640 1 -- 172.21.15.37:0/4134879158 >> v1:172.21.15.146:6789/0 conn(0x7f8c50036690 legacy=0x7f8c50036b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:20.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 1 -- 172.21.15.37:0/4134879158 >> v1:172.21.15.37:6789/0 conn(0x7f8c50038f10 legacy=0x7f8c50039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 1 monclient: found mon.noname-b 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 1 -- 172.21.15.37:0/4134879158 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c500cbc40 con 0x7f8c50006eb0 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 10 monclient: _finish_auth 0 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 1 -- 172.21.15.37:0/4134879158 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8c48001670 con 0x7f8c50006eb0 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 10 monclient: dump: 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:20.751 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c57fff640 5 monclient: authenticate success, global_id 4658 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c57fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 1 -- 172.21.15.37:0/4134879158 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c48001840 con 0x7f8c50006eb0 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c57fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c57fff640 4 set_mon_vals no callback set 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 1 -- 172.21.15.37:0/4134879158 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8c48001c90 con 0x7f8c50006eb0 2024-03-29T11:45:20.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.743+0000 7f8c55ffb640 10 monclient: dump: 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.744+0000 7f8c57fff640 10 monclient: shutdown 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.744+0000 7f8c57fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.744+0000 7f8c57fff640 1 -- 172.21.15.37:0/4134879158 >> v1:172.21.15.37:6790/0 conn(0x7f8c50006eb0 legacy=0x7f8c50031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.744+0000 7f8c57fff640 1 -- 172.21.15.37:0/4134879158 shutdown_connections 2024-03-29T11:45:20.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.744+0000 7f8c57fff640 1 -- 172.21.15.37:0/4134879158 >> 172.21.15.37:0/4134879158 conn(0x7f8c50005c20 msgr2=0x7f8c50006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:20.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.747+0000 7f8c57fff640 1 -- 172.21.15.37:0/4134879158 shutdown_connections 2024-03-29T11:45:20.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.747+0000 7f8c57fff640 1 -- 172.21.15.37:0/4134879158 wait complete. 2024-03-29T11:45:20.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.748+0000 7f8c57fff640 10 monclient: build_initial_monmap 2024-03-29T11:45:20.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.748+0000 7f8c57fff640 10 monclient: monmap: 2024-03-29T11:45:20.756 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:20.756 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:20.756 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:45:20.756 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:45:20.756 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:20.756 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:20.756 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.748+0000 7f8c57fff640 1 librados: starting msgr at 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.748+0000 7f8c57fff640 1 librados: starting objecter 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.749+0000 7f8c57fff640 1 Processor -- start 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.749+0000 7f8c57fff640 1 -- start start 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.749+0000 7f8c57fff640 1 librados: setting wanted keys 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.749+0000 7f8c57fff640 1 librados: calling monclient init 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.749+0000 7f8c57fff640 10 monclient: init 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.750+0000 7f8c57fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.750+0000 7f8c57fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.750+0000 7f8c57fff640 10 monclient(hunting): picked mon.noname-b con 0x7f8c50036690 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.750+0000 7f8c57fff640 10 monclient(hunting): picked mon.noname-a con 0x7f8c50038f10 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:20.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.750+0000 7f8c57fff640 10 monclient(hunting): picked mon.noname-c con 0x7f8c500cf280 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:20.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.750+0000 7f8c57fff640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5000a930 con 0x7f8c50038f10 2024-03-29T11:45:20.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.750+0000 7f8c57fff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5000ac10 con 0x7f8c50036690 2024-03-29T11:45:20.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.750+0000 7f8c57fff640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5000aef0 con 0x7f8c500cf280 2024-03-29T11:45:20.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.750+0000 7f8c57fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:20.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.750+0000 7f8c57fff640 10 monclient(hunting): authenticate will time out at 1500.199109s 2024-03-29T11:45:20.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.754+0000 7f8c567fc640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8c50038f10 0x7f8c500cdf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:57105/0 (socket says 172.21.15.37:57105) 2024-03-29T11:45:20.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.754+0000 7f8c567fc640 1 -- 172.21.15.37:0/2424252310 learned_addr learned my addr 172.21.15.37:0/2424252310 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:20.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.755+0000 7f8c3f7fe640 10 client.?.objecter ms_handle_connect 0x7f8c50038f10 2024-03-29T11:45:20.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.755+0000 7f8c3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:20.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.755+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353404456 0 0) 0x7f8c5000a930 con 0x7f8c50038f10 2024-03-29T11:45:20.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.755+0000 7f8c3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:20.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.755+0000 7f8c3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:20.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.755+0000 7f8c3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:20.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.755+0000 7f8c3f7fe640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:45:20.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.755+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c44004eb0 con 0x7f8c50038f10 2024-03-29T11:45:20.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 client.?.objecter ms_handle_connect 0x7f8c50036690 2024-03-29T11:45:20.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:20.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 client.?.objecter ms_handle_connect 0x7f8c500cf280 2024-03-29T11:45:20.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:20.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 138753745 0 0) 0x7f8c44004eb0 con 0x7f8c50038f10 2024-03-29T11:45:20.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c5000a930 con 0x7f8c50038f10 2024-03-29T11:45:20.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733183356 0 0) 0x7f8c5000aef0 con 0x7f8c500cf280 2024-03-29T11:45:20.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:20.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:20.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:20.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:45:20.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c44004eb0 con 0x7f8c500cf280 2024-03-29T11:45:20.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355826282 0 0) 0x7f8c5000ac10 con 0x7f8c50036690 2024-03-29T11:45:20.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:20.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:20.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:20.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:45:20.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c5000aef0 con 0x7f8c50036690 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8c5804fdd0 con 0x7f8c50038f10 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 monclient(hunting): dump: 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:20.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.757+0000 7f8c3f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 595210577 0 0) 0x7f8c44004eb0 con 0x7f8c500cf280 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c5000ac10 con 0x7f8c500cf280 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2525936921 0 0) 0x7f8c5000aef0 con 0x7f8c50036690 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c44004eb0 con 0x7f8c50036690 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4020309533 0 0) 0x7f8c5000a930 con 0x7f8c50038f10 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 >> v1:172.21.15.37:6790/0 conn(0x7f8c500cf280 legacy=0x7f8c500c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 >> v1:172.21.15.146:6789/0 conn(0x7f8c50036690 legacy=0x7f8c5000b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 1 monclient: found mon.a 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c500cfcc0 con 0x7f8c50038f10 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:45:20.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.758+0000 7f8c3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:20.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c57fff640 5 monclient: authenticate success, global_id 4581 2024-03-29T11:45:20.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c57fff640 10 monclient: _renew_subs 2024-03-29T11:45:20.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c57fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:20.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c57fff640 1 -- 172.21.15.37:0/2424252310 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c500c1f70 con 0x7f8c50038f10 2024-03-29T11:45:20.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c57fff640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:20.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c57fff640 10 monclient: _renew_subs 2024-03-29T11:45:20.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c57fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:20.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c57fff640 1 -- 172.21.15.37:0/2424252310 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c500c2400 con 0x7f8c50038f10 2024-03-29T11:45:20.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c57fff640 1 librados: init done 2024-03-29T11:45:20.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c5804f400 con 0x7f8c50038f10 2024-03-29T11:45:20.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:20.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f8c58075710 con 0x7f8c50038f10 2024-03-29T11:45:20.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:20.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c3f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:20.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c3f7fe640 10 monclient: dump: 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.759+0000 7f8c557fa640 4 set_mon_vals no callback set 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.761+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f8c58067bf0 con 0x7f8c50038f10 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.761+0000 7f8c57fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.761+0000 7f8c57fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:20.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.761+0000 7f8c57fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:20.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.761+0000 7f8c57fff640 1 -- 172.21.15.37:0/2424252310 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c38005080 con 0x7f8c50038f10 2024-03-29T11:45:20.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.762+0000 7f8c56ffd640 10 monclient: get_auth_request con 0x7f8c4406e010 auth_method 0 2024-03-29T11:45:20.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.762+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2864318804 0 0) 0x7f8c580ff4a0 con 0x7f8c50038f10 2024-03-29T11:45:20.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.762+0000 7f8c3f7fe640 10 client.4581.objecter ms_dispatch 0x7f8c58000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:20.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.762+0000 7f8c3f7fe640 3 client.4581.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:20.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.762+0000 7f8c3f7fe640 3 client.4581.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:20.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.762+0000 7f8c3f7fe640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-29T11:45:20.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.762+0000 7f8c3f7fe640 10 client.4581.objecter ms_handle_connect 0x7f8c4406e010 2024-03-29T11:45:20.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.770+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c580cad80 con 0x7f8c50038f10 2024-03-29T11:45:20.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.770+0000 7f8c3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:20.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:20.770+0000 7f8c3f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:21.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.119+0000 7f8c57fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:21.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.119+0000 7f8c57fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:21.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.119+0000 7f8c57fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:21.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.119+0000 7f8c57fff640 1 -- 172.21.15.37:0/2424252310 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c38005330 con 0x7f8c50038f10 2024-03-29T11:45:21.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.120+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/2424252310 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1684400214) 0x7f8c580cccb0 con 0x7f8c50038f10 2024-03-29T11:45:21.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.120+0000 7f8c3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:21.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.120+0000 7f8c3f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:21.125 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:45:21.125 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:45:04.373638+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:04.373638+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:21.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.124+0000 7f8c57fff640 10 librados: watch_flush enter 2024-03-29T11:45:21.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.125+0000 7f8c57fff640 10 librados: watch_flush exit 2024-03-29T11:45:21.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.125+0000 7f8c57fff640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-29T11:45:21.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.125+0000 7f8c57fff640 10 client.4581.objecter successfully canceled tick 2024-03-29T11:45:21.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.126+0000 7f8c57fff640 1 -- 172.21.15.37:0/2424252310 >> v1:172.21.15.146:6816/2899149139 conn(0x7f8c4406e010 legacy=0x7f8c440704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:21.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.126+0000 7f8c57fff640 10 monclient: shutdown 2024-03-29T11:45:21.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.126+0000 7f8c57fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:21.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.126+0000 7f8c57fff640 1 -- 172.21.15.37:0/2424252310 >> v1:172.21.15.37:6789/0 conn(0x7f8c50038f10 legacy=0x7f8c500cdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:21.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.126+0000 7f8c57fff640 1 -- 172.21.15.37:0/2424252310 shutdown_connections 2024-03-29T11:45:21.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.127+0000 7f8c57fff640 1 -- 172.21.15.37:0/2424252310 >> 172.21.15.37:0/2424252310 conn(0x7f8c50005c20 msgr2=0x7f8c50006000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:21.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.127+0000 7f8c57fff640 1 -- 172.21.15.37:0/2424252310 shutdown_connections 2024-03-29T11:45:21.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.127+0000 7f8c57fff640 1 -- 172.21.15.37:0/2424252310 wait complete. 2024-03-29T11:45:21.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:21.127+0000 7f8c57fff640 1 librados: shutdown 2024-03-29T11:45:21.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:27.956 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:28.116 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.111+0000 7f5f569e8640 10 monclient: get_monmap_and_config 2024-03-29T11:45:28.116 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.111+0000 7f5f569e8640 10 monclient: build_initial_monmap 2024-03-29T11:45:28.116 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.111+0000 7f5f569e8640 10 monclient: monmap: 2024-03-29T11:45:28.116 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:28.116 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:28.116 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:45:28.112473+0000 2024-03-29T11:45:28.116 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:45:28.112473+0000 2024-03-29T11:45:28.116 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:28.116 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:28.116 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:28.116 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:45:28.117 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:45:28.117 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:28.117 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.112+0000 7f5f569e8640 1 Processor -- start 2024-03-29T11:45:28.117 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.113+0000 7f5f569e8640 1 -- start start 2024-03-29T11:45:28.117 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.113+0000 7f5f569e8640 10 monclient: init 2024-03-29T11:45:28.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.114+0000 7f5f569e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:28.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.114+0000 7f5f569e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:45:28.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.114+0000 7f5f569e8640 10 monclient(hunting): picked mon.noname-c con 0x7f5f480389e0 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:28.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.114+0000 7f5f569e8640 10 monclient(hunting): picked mon.noname-b con 0x7f5f48038dc0 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:28.119 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.114+0000 7f5f551e5640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f5f48038dc0 0x7f5f480cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:44450/0 (socket says 172.21.15.37:44450) 2024-03-29T11:45:28.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.118+0000 7f5f551e5640 1 -- 172.21.15.37:0/2575041245 learned_addr learned my addr 172.21.15.37:0/2575041245 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:28.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.118+0000 7f5f569e8640 10 monclient(hunting): picked mon.noname-a con 0x7f5f48030660 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:28.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.118+0000 7f5f569e8640 1 -- 172.21.15.37:0/2575041245 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f480c54c0 con 0x7f5f48030660 2024-03-29T11:45:28.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.118+0000 7f5f569e8640 1 -- 172.21.15.37:0/2575041245 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f48036430 con 0x7f5f480389e0 2024-03-29T11:45:28.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.118+0000 7f5f569e8640 1 -- 172.21.15.37:0/2575041245 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f48039840 con 0x7f5f48038dc0 2024-03-29T11:45:28.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.118+0000 7f5f569e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:28.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.118+0000 7f5f569e8640 10 monclient(hunting): authenticate will time out at 1507.567619s 2024-03-29T11:45:28.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.119+0000 7f5f549e4640 1 -- 172.21.15.37:0/2575041245 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426210882 0 0) 0x7f5f480c54c0 con 0x7f5f48030660 2024-03-29T11:45:28.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.119+0000 7f5f549e4640 10 monclient(hunting): _negotiate 2024-03-29T11:45:28.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.119+0000 7f5f549e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:28.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.119+0000 7f5f549e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:28.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.119+0000 7f5f549e4640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:45:28.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.119+0000 7f5f549e4640 1 -- 172.21.15.37:0/2575041245 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f3c003520 con 0x7f5f48030660 2024-03-29T11:45:28.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.119+0000 7f5f549e4640 1 -- 172.21.15.37:0/2575041245 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 265130278 0 0) 0x7f5f3c003520 con 0x7f5f48030660 2024-03-29T11:45:28.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.119+0000 7f5f549e4640 1 -- 172.21.15.37:0/2575041245 >> v1:172.21.15.37:6790/0 conn(0x7f5f48038dc0 legacy=0x7f5f480cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:28.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.119+0000 7f5f549e4640 1 -- 172.21.15.37:0/2575041245 >> v1:172.21.15.146:6789/0 conn(0x7f5f480389e0 legacy=0x7f5f48033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:28.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.121+0000 7f5f549e4640 10 monclient: _finish_hunting 0 2024-03-29T11:45:28.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.121+0000 7f5f549e4640 1 monclient: found mon.noname-a 2024-03-29T11:45:28.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.121+0000 7f5f549e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:28.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.121+0000 7f5f549e4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-29T11:45:28.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.121+0000 7f5f549e4640 1 -- 172.21.15.37:0/2575041245 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f48035690 con 0x7f5f48030660 2024-03-29T11:45:28.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.121+0000 7f5f549e4640 10 monclient: _finish_auth 0 2024-03-29T11:45:28.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.121+0000 7f5f549e4640 10 monclient: _check_auth_tickets 2024-03-29T11:45:28.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.121+0000 7f5f549e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.121+0000 7f5f549e4640 1 -- 172.21.15.37:0/2575041245 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5f5004f730 con 0x7f5f48030660 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.121+0000 7f5f549e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.121+0000 7f5f549e4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.121+0000 7f5f549e4640 10 monclient: dump: 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:28.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.121+0000 7f5f569e8640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:45:28.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.121+0000 7f5f569e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:28.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.123+0000 7f5f549e4640 1 -- 172.21.15.37:0/2575041245 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f500008d0 con 0x7f5f48030660 2024-03-29T11:45:28.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.123+0000 7f5f549e4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:28.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.123+0000 7f5f549e4640 1 -- 172.21.15.37:0/2575041245 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5f500758b0 con 0x7f5f48030660 2024-03-29T11:45:28.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.123+0000 7f5f549e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:28.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.123+0000 7f5f549e4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:28.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.123+0000 7f5f549e4640 10 monclient: dump: 2024-03-29T11:45:28.127 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:28.127 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:28.128 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:28.128 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:28.128 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:28.128 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:28.128 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:28.128 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:28.128 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:28.128 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:28.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.124+0000 7f5f569e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:28.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.124+0000 7f5f569e8640 4 set_mon_vals no callback set 2024-03-29T11:45:28.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.125+0000 7f5f569e8640 10 monclient: shutdown 2024-03-29T11:45:28.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.125+0000 7f5f569e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:28.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.125+0000 7f5f569e8640 1 -- 172.21.15.37:0/2575041245 >> v1:172.21.15.37:6789/0 conn(0x7f5f48030660 legacy=0x7f5f480d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:28.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.125+0000 7f5f569e8640 1 -- 172.21.15.37:0/2575041245 shutdown_connections 2024-03-29T11:45:28.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.125+0000 7f5f569e8640 1 -- 172.21.15.37:0/2575041245 >> 172.21.15.37:0/2575041245 conn(0x7f5f48005c00 msgr2=0x7f5f48006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:28.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.127+0000 7f5f569e8640 1 -- 172.21.15.37:0/2575041245 shutdown_connections 2024-03-29T11:45:28.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.127+0000 7f5f569e8640 1 -- 172.21.15.37:0/2575041245 wait complete. 2024-03-29T11:45:28.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.128+0000 7f5f569e8640 10 monclient: build_initial_monmap 2024-03-29T11:45:28.133 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.128+0000 7f5f569e8640 10 monclient: monmap: 2024-03-29T11:45:28.133 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:28.133 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:28.133 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:45:28.133 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:45:28.133 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:28.133 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:28.133 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:28.133 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:45:28.133 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:45:28.133 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:28.134 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.129+0000 7f5f569e8640 1 librados: starting msgr at 2024-03-29T11:45:28.134 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.129+0000 7f5f569e8640 1 librados: starting objecter 2024-03-29T11:45:28.134 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.129+0000 7f5f569e8640 1 Processor -- start 2024-03-29T11:45:28.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.130+0000 7f5f569e8640 1 -- start start 2024-03-29T11:45:28.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.130+0000 7f5f569e8640 1 librados: setting wanted keys 2024-03-29T11:45:28.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.130+0000 7f5f569e8640 1 librados: calling monclient init 2024-03-29T11:45:28.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.130+0000 7f5f569e8640 10 monclient: init 2024-03-29T11:45:28.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.131+0000 7f5f569e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:28.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.131+0000 7f5f569e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:28.138 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.134+0000 7f5f569e8640 10 monclient(hunting): picked mon.noname-a con 0x7f5f48030660 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:28.138 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.134+0000 7f5f569e8640 10 monclient(hunting): picked mon.noname-c con 0x7f5f480389e0 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:28.139 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.134+0000 7f5f551e5640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f5f480389e0 0x7f5f4802f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:48726/0 (socket says 172.21.15.37:48726) 2024-03-29T11:45:28.139 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.135+0000 7f5f551e5640 1 -- 172.21.15.37:0/1082416243 learned_addr learned my addr 172.21.15.37:0/1082416243 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:28.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.135+0000 7f5f569e8640 10 monclient(hunting): picked mon.noname-b con 0x7f5f4802d3c0 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:28.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.138+0000 7f5f569e8640 1 -- 172.21.15.37:0/1082416243 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4802e260 con 0x7f5f48030660 2024-03-29T11:45:28.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.138+0000 7f5f467fc640 10 client.?.objecter ms_handle_connect 0x7f5f48030660 2024-03-29T11:45:28.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.138+0000 7f5f467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:28.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.139+0000 7f5f569e8640 1 -- 172.21.15.37:0/1082416243 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4802e540 con 0x7f5f4802d3c0 2024-03-29T11:45:28.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.139+0000 7f5f467fc640 10 client.?.objecter ms_handle_connect 0x7f5f480389e0 2024-03-29T11:45:28.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.139+0000 7f5f467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:28.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.139+0000 7f5f467fc640 10 client.?.objecter ms_handle_connect 0x7f5f4802d3c0 2024-03-29T11:45:28.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.139+0000 7f5f467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:28.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.139+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264088616 0 0) 0x7f5f4802e260 con 0x7f5f48030660 2024-03-29T11:45:28.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.139+0000 7f5f569e8640 1 -- 172.21.15.37:0/1082416243 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4802e820 con 0x7f5f480389e0 2024-03-29T11:45:28.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.139+0000 7f5f569e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:28.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.139+0000 7f5f569e8640 10 monclient(hunting): authenticate will time out at 1507.588658s 2024-03-29T11:45:28.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.140+0000 7f5f467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:28.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.140+0000 7f5f467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:28.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.140+0000 7f5f467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:28.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.140+0000 7f5f467fc640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:45:28.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.140+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f34004eb0 con 0x7f5f48030660 2024-03-29T11:45:28.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.140+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613381516 0 0) 0x7f5f4802e540 con 0x7f5f4802d3c0 2024-03-29T11:45:28.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.140+0000 7f5f467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:28.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.140+0000 7f5f467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:28.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.140+0000 7f5f467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:28.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.140+0000 7f5f467fc640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:45:28.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.140+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f4802e260 con 0x7f5f4802d3c0 2024-03-29T11:45:28.151 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.141+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583852365 0 0) 0x7f5f4802e820 con 0x7f5f480389e0 2024-03-29T11:45:28.151 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.141+0000 7f5f467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:28.151 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.141+0000 7f5f467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:28.151 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.142+0000 7f5f467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:28.151 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.142+0000 7f5f467fc640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:45:28.151 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.142+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f4802e540 con 0x7f5f480389e0 2024-03-29T11:45:28.151 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.142+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3161100395 0 0) 0x7f5f34004eb0 con 0x7f5f48030660 2024-03-29T11:45:28.151 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.142+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f4802e820 con 0x7f5f48030660 2024-03-29T11:45:28.151 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.142+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2310583781 0 0) 0x7f5f4802e260 con 0x7f5f4802d3c0 2024-03-29T11:45:28.152 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.142+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f34004eb0 con 0x7f5f4802d3c0 2024-03-29T11:45:28.152 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.142+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5f40001870 con 0x7f5f48030660 2024-03-29T11:45:28.152 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.142+0000 7f5f467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:28.152 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.142+0000 7f5f467fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:28.152 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.143+0000 7f5f467fc640 10 monclient(hunting): dump: 2024-03-29T11:45:28.152 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:28.152 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:28.152 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:28.152 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:28.152 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:28.152 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:28.152 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:28.152 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:28.152 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.143+0000 7f5f467fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.143+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5f5004f400 con 0x7f5f4802d3c0 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.143+0000 7f5f467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.143+0000 7f5f467fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.143+0000 7f5f467fc640 10 monclient(hunting): dump: 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:28.153 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:28.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.143+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2670926371 0 0) 0x7f5f4802e540 con 0x7f5f480389e0 2024-03-29T11:45:28.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.143+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f4802e260 con 0x7f5f480389e0 2024-03-29T11:45:28.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.143+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2070706260 0 0) 0x7f5f34004eb0 con 0x7f5f4802d3c0 2024-03-29T11:45:28.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.143+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 >> v1:172.21.15.37:6790/0 conn(0x7f5f480389e0 legacy=0x7f5f4802f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:28.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.143+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 >> v1:172.21.15.37:6789/0 conn(0x7f5f48030660 legacy=0x7f5f480bffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:28.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f467fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:28.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f467fc640 1 monclient: found mon.b 2024-03-29T11:45:28.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:28.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f467fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:28.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4802eb00 con 0x7f5f4802d3c0 2024-03-29T11:45:28.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f467fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:28.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f467fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:28.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:28.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f5004f730 con 0x7f5f4802d3c0 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f569e8640 5 monclient: authenticate success, global_id 4537 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f569e8640 10 monclient: _renew_subs 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f569e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f569e8640 1 -- 172.21.15.37:0/1082416243 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f480c0a90 con 0x7f5f4802d3c0 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f569e8640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f569e8640 10 monclient: _renew_subs 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.144+0000 7f5f569e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.145+0000 7f5f569e8640 1 -- 172.21.15.37:0/1082416243 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f480c06e0 con 0x7f5f4802d3c0 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.145+0000 7f5f569e8640 1 librados: init done 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.145+0000 7f5f467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.145+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5f50067d90 con 0x7f5f4802d3c0 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.145+0000 7f5f467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.145+0000 7f5f467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.145+0000 7f5f477fe640 4 set_mon_vals no callback set 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.145+0000 7f5f467fc640 10 monclient: dump: 2024-03-29T11:45:28.155 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:28.156 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:28.156 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:28.156 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:28.156 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:28.156 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:28.156 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:28.156 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:28.156 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:28.156 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:28.156 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.146+0000 7f5f569e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:28.156 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.146+0000 7f5f569e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:28.156 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.146+0000 7f5f569e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:28.156 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.147+0000 7f5f569e8640 1 -- 172.21.15.37:0/1082416243 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f38005080 con 0x7f5f4802d3c0 2024-03-29T11:45:28.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.147+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f5f50067ef0 con 0x7f5f4802d3c0 2024-03-29T11:45:28.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.148+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2864318804 0 0) 0x7f5f500fdf20 con 0x7f5f4802d3c0 2024-03-29T11:45:28.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.148+0000 7f5f467fc640 10 client.4537.objecter ms_dispatch 0x7f5f50000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:28.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.148+0000 7f5f559e6640 10 monclient: get_auth_request con 0x7f5f3406dfa0 auth_method 0 2024-03-29T11:45:28.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.149+0000 7f5f467fc640 3 client.4537.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:28.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.149+0000 7f5f467fc640 3 client.4537.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:28.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.149+0000 7f5f467fc640 20 client.4537.objecter dump_active .. 0 homeless 2024-03-29T11:45:28.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.151+0000 7f5f467fc640 10 client.4537.objecter ms_handle_connect 0x7f5f3406dfa0 2024-03-29T11:45:28.162 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.154+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f500c9800 con 0x7f5f4802d3c0 2024-03-29T11:45:28.162 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.154+0000 7f5f467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:28.162 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.154+0000 7f5f467fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.506+0000 7f5f569e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.506+0000 7f5f569e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.506+0000 7f5f569e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:28.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.506+0000 7f5f569e8640 1 -- 172.21.15.37:0/1082416243 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f38005330 con 0x7f5f4802d3c0 2024-03-29T11:45:28.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.506+0000 7f5f467fc640 1 -- 172.21.15.37:0/1082416243 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1684400214) 0x7f5f500cb730 con 0x7f5f4802d3c0 2024-03-29T11:45:28.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.506+0000 7f5f467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:28.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.506+0000 7f5f467fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:28.513 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:45:28.513 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:45:04.373638+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:04.373638+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:28.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.512+0000 7f5f569e8640 10 librados: watch_flush enter 2024-03-29T11:45:28.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.515+0000 7f5f569e8640 10 librados: watch_flush exit 2024-03-29T11:45:28.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.515+0000 7f5f569e8640 20 client.4537.objecter shutdown clearing up homeless session... 2024-03-29T11:45:28.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.515+0000 7f5f569e8640 10 client.4537.objecter successfully canceled tick 2024-03-29T11:45:28.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.515+0000 7f5f569e8640 1 -- 172.21.15.37:0/1082416243 >> v1:172.21.15.146:6816/2899149139 conn(0x7f5f3406dfa0 legacy=0x7f5f34070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:28.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.515+0000 7f5f569e8640 10 monclient: shutdown 2024-03-29T11:45:28.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.515+0000 7f5f569e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:28.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.515+0000 7f5f569e8640 1 -- 172.21.15.37:0/1082416243 >> v1:172.21.15.146:6789/0 conn(0x7f5f4802d3c0 legacy=0x7f5f4816c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:28.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.515+0000 7f5f569e8640 1 -- 172.21.15.37:0/1082416243 shutdown_connections 2024-03-29T11:45:28.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.515+0000 7f5f569e8640 1 -- 172.21.15.37:0/1082416243 >> 172.21.15.37:0/1082416243 conn(0x7f5f48005c00 msgr2=0x7f5f48034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:28.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.515+0000 7f5f569e8640 1 -- 172.21.15.37:0/1082416243 shutdown_connections 2024-03-29T11:45:28.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.515+0000 7f5f569e8640 1 -- 172.21.15.37:0/1082416243 wait complete. 2024-03-29T11:45:28.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:28.516+0000 7f5f569e8640 1 librados: shutdown 2024-03-29T11:45:28.538 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:35.346 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:35.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.503+0000 7fea38c0f640 10 monclient: get_monmap_and_config 2024-03-29T11:45:35.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.503+0000 7fea38c0f640 10 monclient: build_initial_monmap 2024-03-29T11:45:35.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.503+0000 7fea38c0f640 10 monclient: monmap: 2024-03-29T11:45:35.508 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:35.508 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:35.508 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:45:35.504106+0000 2024-03-29T11:45:35.508 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:45:35.504106+0000 2024-03-29T11:45:35.508 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:35.508 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:35.509 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:35.509 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:45:35.509 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:45:35.509 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:35.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.505+0000 7fea38c0f640 1 Processor -- start 2024-03-29T11:45:35.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.505+0000 7fea38c0f640 1 -- start start 2024-03-29T11:45:35.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.505+0000 7fea38c0f640 10 monclient: init 2024-03-29T11:45:35.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.507+0000 7fea38c0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:35.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.507+0000 7fea38c0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:45:35.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.507+0000 7fea38c0f640 10 monclient(hunting): picked mon.noname-c con 0x7fea340b1f60 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:35.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.507+0000 7fea38c0f640 10 monclient(hunting): picked mon.noname-b con 0x7fea34071eb0 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:35.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.507+0000 7fea38c0f640 10 monclient(hunting): picked mon.noname-a con 0x7fea34143d10 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:35.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.507+0000 7fea38c0f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3406ef80 con 0x7fea34143d10 2024-03-29T11:45:35.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.507+0000 7fea38c0f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3406f260 con 0x7fea340b1f60 2024-03-29T11:45:35.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.507+0000 7fea38c0f640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea340b40c0 con 0x7fea34071eb0 2024-03-29T11:45:35.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.507+0000 7fea38c0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:35.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.507+0000 7fea38c0f640 10 monclient(hunting): authenticate will time out at 1514.956256s 2024-03-29T11:45:35.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.509+0000 7fea3259c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fea340b1f60 0x7fea34143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:56948/0 (socket says 172.21.15.37:56948) 2024-03-29T11:45:35.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.509+0000 7fea3259c640 1 -- 172.21.15.37:0/3279336934 learned_addr learned my addr 172.21.15.37:0/3279336934 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:35.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.511+0000 7fea3159a640 1 -- 172.21.15.37:0/3279336934 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407056602 0 0) 0x7fea3406f260 con 0x7fea340b1f60 2024-03-29T11:45:35.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.511+0000 7fea3159a640 10 monclient(hunting): _negotiate 2024-03-29T11:45:35.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.511+0000 7fea3159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:35.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.511+0000 7fea3159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:35.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.511+0000 7fea3159a640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:45:35.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.511+0000 7fea3159a640 1 -- 172.21.15.37:0/3279336934 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea1c003520 con 0x7fea340b1f60 2024-03-29T11:45:35.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.511+0000 7fea3159a640 1 -- 172.21.15.37:0/3279336934 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879635145 0 0) 0x7fea3406ef80 con 0x7fea34143d10 2024-03-29T11:45:35.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.511+0000 7fea3159a640 10 monclient(hunting): _negotiate 2024-03-29T11:45:35.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.511+0000 7fea3159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:35.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.511+0000 7fea3159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:35.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.511+0000 7fea3159a640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:45:35.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.511+0000 7fea3159a640 1 -- 172.21.15.37:0/3279336934 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea1c003840 con 0x7fea34143d10 2024-03-29T11:45:35.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 1 -- 172.21.15.37:0/3279336934 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2880637937 0 0) 0x7fea1c003520 con 0x7fea340b1f60 2024-03-29T11:45:35.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 1 -- 172.21.15.37:0/3279336934 >> v1:172.21.15.37:6790/0 conn(0x7fea34071eb0 legacy=0x7fea34146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:35.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 1 -- 172.21.15.37:0/3279336934 >> v1:172.21.15.37:6789/0 conn(0x7fea34143d10 legacy=0x7fea34146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:35.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 10 monclient: _finish_hunting 0 2024-03-29T11:45:35.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 1 monclient: found mon.noname-c 2024-03-29T11:45:35.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:35.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-29T11:45:35.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 1 -- 172.21.15.37:0/3279336934 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea34155d70 con 0x7fea340b1f60 2024-03-29T11:45:35.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 10 monclient: _finish_auth 0 2024-03-29T11:45:35.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 10 monclient: _check_auth_tickets 2024-03-29T11:45:35.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:35.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 1 -- 172.21.15.37:0/3279336934 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fea28001670 con 0x7fea340b1f60 2024-03-29T11:45:35.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:35.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:35.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 10 monclient: dump: 2024-03-29T11:45:35.520 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:35.520 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:35.520 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:35.520 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:35.520 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:35.520 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:35.521 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:35.521 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:35.521 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:35.521 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:35.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea38c0f640 5 monclient: authenticate success, global_id 4540 2024-03-29T11:45:35.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea38c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:35.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 1 -- 172.21.15.37:0/3279336934 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea28001850 con 0x7fea340b1f60 2024-03-29T11:45:35.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea3159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:35.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea38c0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:35.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.513+0000 7fea38c0f640 4 set_mon_vals no callback set 2024-03-29T11:45:35.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.514+0000 7fea38c0f640 10 monclient: shutdown 2024-03-29T11:45:35.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.514+0000 7fea38c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:35.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.514+0000 7fea38c0f640 1 -- 172.21.15.37:0/3279336934 >> v1:172.21.15.146:6789/0 conn(0x7fea340b1f60 legacy=0x7fea34143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:35.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.514+0000 7fea3159a640 1 -- 172.21.15.37:0/3279336934 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fea28001670 con 0x7fea340b1f60 2024-03-29T11:45:35.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.514+0000 7fea3159a640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:45:35.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.514+0000 7fea38c0f640 1 -- 172.21.15.37:0/3279336934 shutdown_connections 2024-03-29T11:45:35.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.514+0000 7fea38c0f640 1 -- 172.21.15.37:0/3279336934 >> 172.21.15.37:0/3279336934 conn(0x7fea34066a00 msgr2=0x7fea34066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:35.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.514+0000 7fea38c0f640 1 -- 172.21.15.37:0/3279336934 shutdown_connections 2024-03-29T11:45:35.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.514+0000 7fea38c0f640 1 -- 172.21.15.37:0/3279336934 wait complete. 2024-03-29T11:45:35.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.516+0000 7fea38c0f640 10 monclient: build_initial_monmap 2024-03-29T11:45:35.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.516+0000 7fea38c0f640 10 monclient: monmap: 2024-03-29T11:45:35.524 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:35.524 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:35.524 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:45:35.524 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:45:35.524 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:35.525 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:35.525 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:35.525 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:45:35.525 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:45:35.525 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:35.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.516+0000 7fea38c0f640 1 librados: starting msgr at 2024-03-29T11:45:35.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.516+0000 7fea38c0f640 1 librados: starting objecter 2024-03-29T11:45:35.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.516+0000 7fea38c0f640 1 Processor -- start 2024-03-29T11:45:35.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.516+0000 7fea38c0f640 1 -- start start 2024-03-29T11:45:35.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.516+0000 7fea38c0f640 1 librados: setting wanted keys 2024-03-29T11:45:35.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.516+0000 7fea38c0f640 1 librados: calling monclient init 2024-03-29T11:45:35.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.516+0000 7fea38c0f640 10 monclient: init 2024-03-29T11:45:35.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.517+0000 7fea38c0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:35.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.517+0000 7fea38c0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:35.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.517+0000 7fea38c0f640 10 monclient(hunting): picked mon.noname-c con 0x7fea340b1f60 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:35.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.517+0000 7fea38c0f640 10 monclient(hunting): picked mon.noname-a con 0x7fea34143d10 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:35.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.517+0000 7fea38c0f640 10 monclient(hunting): picked mon.noname-b con 0x7fea3413bce0 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:35.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.517+0000 7fea38c0f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3413d8f0 con 0x7fea34143d10 2024-03-29T11:45:35.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.517+0000 7fea38c0f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3413db60 con 0x7fea3413bce0 2024-03-29T11:45:35.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.517+0000 7fea38c0f640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3413c290 con 0x7fea340b1f60 2024-03-29T11:45:35.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.517+0000 7fea38c0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:35.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.517+0000 7fea38c0f640 10 monclient(hunting): authenticate will time out at 1514.966691s 2024-03-29T11:45:35.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.520+0000 7fea32d9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fea3413bce0 0x7fea341e7070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:58316/0 (socket says 172.21.15.37:58316) 2024-03-29T11:45:35.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.520+0000 7fea32d9d640 1 -- 172.21.15.37:0/573512534 learned_addr learned my addr 172.21.15.37:0/573512534 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:35.534 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.520+0000 7fea22ffd640 10 client.?.objecter ms_handle_connect 0x7fea3413bce0 2024-03-29T11:45:35.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.520+0000 7fea22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:35.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.522+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869229058 0 0) 0x7fea3413db60 con 0x7fea3413bce0 2024-03-29T11:45:35.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.522+0000 7fea22ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:35.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.522+0000 7fea22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:35.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.522+0000 7fea22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:35.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.522+0000 7fea22ffd640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:45:35.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.522+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea14004eb0 con 0x7fea3413bce0 2024-03-29T11:45:35.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.522+0000 7fea22ffd640 10 client.?.objecter ms_handle_connect 0x7fea34143d10 2024-03-29T11:45:35.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.522+0000 7fea22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:35.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.522+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4034694837 0 0) 0x7fea14004eb0 con 0x7fea3413bce0 2024-03-29T11:45:35.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.522+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea3413db60 con 0x7fea3413bce0 2024-03-29T11:45:35.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.522+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fea2c001c10 con 0x7fea3413bce0 2024-03-29T11:45:35.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.522+0000 7fea22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:35.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:35.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 10 monclient(hunting): dump: 2024-03-29T11:45:35.536 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:35.536 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:35.536 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:35.536 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:35.537 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:35.537 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:35.537 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:35.537 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:35.537 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:35.537 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:35.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:35.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223877003 0 0) 0x7fea3413d8f0 con 0x7fea34143d10 2024-03-29T11:45:35.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:35.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:35.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:35.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:45:35.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea14004eb0 con 0x7fea34143d10 2024-03-29T11:45:35.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 403088278 0 0) 0x7fea3413db60 con 0x7fea3413bce0 2024-03-29T11:45:35.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 >> v1:172.21.15.37:6790/0 conn(0x7fea340b1f60 legacy=0x7fea34138420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:35.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 >> v1:172.21.15.37:6789/0 conn(0x7fea34143d10 legacy=0x7fea34140360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:35.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:35.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 1 monclient: found mon.b 2024-03-29T11:45:35.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:35.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:35.538 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3413ba10 con 0x7fea3413bce0 2024-03-29T11:45:35.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 10 monclient: _finish_auth 0 2024-03-29T11:45:35.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:35.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:35.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea38c0f640 5 monclient: authenticate success, global_id 4543 2024-03-29T11:45:35.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea38c0f640 10 monclient: _renew_subs 2024-03-29T11:45:35.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea38c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:35.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.523+0000 7fea38c0f640 1 -- 172.21.15.37:0/573512534 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea34138b30 con 0x7fea3413bce0 2024-03-29T11:45:35.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.524+0000 7fea38c0f640 10 client.4543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:35.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.524+0000 7fea38c0f640 10 monclient: _renew_subs 2024-03-29T11:45:35.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.524+0000 7fea38c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:35.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.524+0000 7fea38c0f640 1 -- 172.21.15.37:0/573512534 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3413c990 con 0x7fea3413bce0 2024-03-29T11:45:35.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.524+0000 7fea38c0f640 1 librados: init done 2024-03-29T11:45:35.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.524+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea2c001de0 con 0x7fea3413bce0 2024-03-29T11:45:35.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.524+0000 7fea22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:35.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.524+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7fea2c003b90 con 0x7fea3413bce0 2024-03-29T11:45:35.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.524+0000 7fea22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:35.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.524+0000 7fea22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:35.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.524+0000 7fea22ffd640 10 monclient: dump: 2024-03-29T11:45:35.540 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:35.540 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:35.540 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:35.540 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:35.540 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:35.541 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:35.541 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:35.541 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:35.541 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:35.541 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:35.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.524+0000 7fea23fff640 4 set_mon_vals no callback set 2024-03-29T11:45:35.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.526+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7fea2c003df0 con 0x7fea3413bce0 2024-03-29T11:45:35.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.526+0000 7fea38c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:35.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.527+0000 7fea38c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:35.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.527+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2864318804 0 0) 0x7fea2c084db0 con 0x7fea3413bce0 2024-03-29T11:45:35.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.527+0000 7fea38c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:35.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.527+0000 7fea22ffd640 10 client.4543.objecter ms_dispatch 0x7fea34000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:35.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.527+0000 7fea22ffd640 3 client.4543.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:35.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.527+0000 7fea22ffd640 3 client.4543.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:35.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.527+0000 7fea22ffd640 20 client.4543.objecter dump_active .. 0 homeless 2024-03-29T11:45:35.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.527+0000 7fea38c0f640 1 -- 172.21.15.37:0/573512534 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea18005080 con 0x7fea3413bce0 2024-03-29T11:45:35.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.531+0000 7fea3259c640 10 monclient: get_auth_request con 0x7fea1406dd70 auth_method 0 2024-03-29T11:45:35.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.531+0000 7fea22ffd640 10 client.4543.objecter ms_handle_connect 0x7fea1406dd70 2024-03-29T11:45:35.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.537+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea2c050690 con 0x7fea3413bce0 2024-03-29T11:45:35.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.537+0000 7fea22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:35.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.537+0000 7fea22ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:35.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.908+0000 7fea38c0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:35.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.908+0000 7fea38c0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:35.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.908+0000 7fea38c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:35.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.908+0000 7fea38c0f640 1 -- 172.21.15.37:0/573512534 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea18005330 con 0x7fea3413bce0 2024-03-29T11:45:35.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.910+0000 7fea22ffd640 1 -- 172.21.15.37:0/573512534 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1684400214) 0x7fea2c0525c0 con 0x7fea3413bce0 2024-03-29T11:45:35.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.910+0000 7fea22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:35.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.910+0000 7fea22ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:35.915 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:45:35.915 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:45:04.373638+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:04.373638+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:35.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.914+0000 7fea38c0f640 10 librados: watch_flush enter 2024-03-29T11:45:35.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.914+0000 7fea38c0f640 10 librados: watch_flush exit 2024-03-29T11:45:35.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.914+0000 7fea38c0f640 20 client.4543.objecter shutdown clearing up homeless session... 2024-03-29T11:45:35.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.914+0000 7fea38c0f640 10 client.4543.objecter successfully canceled tick 2024-03-29T11:45:35.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.916+0000 7fea38c0f640 1 -- 172.21.15.37:0/573512534 >> v1:172.21.15.146:6816/2899149139 conn(0x7fea1406dd70 legacy=0x7fea14070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:35.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.916+0000 7fea38c0f640 10 monclient: shutdown 2024-03-29T11:45:35.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.916+0000 7fea38c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:35.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.916+0000 7fea38c0f640 1 -- 172.21.15.37:0/573512534 >> v1:172.21.15.146:6789/0 conn(0x7fea3413bce0 legacy=0x7fea341e7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:35.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.916+0000 7fea38c0f640 1 -- 172.21.15.37:0/573512534 shutdown_connections 2024-03-29T11:45:35.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.916+0000 7fea38c0f640 1 -- 172.21.15.37:0/573512534 >> 172.21.15.37:0/573512534 conn(0x7fea34066a00 msgr2=0x7fea34066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:35.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.917+0000 7fea38c0f640 1 -- 172.21.15.37:0/573512534 shutdown_connections 2024-03-29T11:45:35.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.917+0000 7fea38c0f640 1 -- 172.21.15.37:0/573512534 wait complete. 2024-03-29T11:45:35.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:35.917+0000 7fea38c0f640 1 librados: shutdown 2024-03-29T11:45:35.939 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:42.746 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:42.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.892+0000 7f12174d0640 10 monclient: get_monmap_and_config 2024-03-29T11:45:42.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.892+0000 7f12174d0640 10 monclient: build_initial_monmap 2024-03-29T11:45:42.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.892+0000 7f12174d0640 10 monclient: monmap: 2024-03-29T11:45:42.898 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:42.898 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:42.898 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:45:42.894384+0000 2024-03-29T11:45:42.898 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:45:42.894384+0000 2024-03-29T11:45:42.898 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:42.898 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:42.898 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:42.898 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:45:42.898 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:45:42.899 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:42.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.893+0000 7f12174d0640 1 Processor -- start 2024-03-29T11:45:42.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.894+0000 7f12174d0640 1 -- start start 2024-03-29T11:45:42.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.894+0000 7f12174d0640 10 monclient: init 2024-03-29T11:45:42.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f12174d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:42.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f12174d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:42.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f12174d0640 10 monclient(hunting): picked mon.noname-a con 0x7f12100b4790 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:42.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f12174d0640 10 monclient(hunting): picked mon.noname-c con 0x7f12100aea90 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:42.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f12174d0640 10 monclient(hunting): picked mon.noname-b con 0x7f12100b39f0 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:42.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f12174d0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12100b0c60 con 0x7f12100b4790 2024-03-29T11:45:42.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f12174d0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12100af020 con 0x7f12100aea90 2024-03-29T11:45:42.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f12174d0640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121013fbd0 con 0x7f12100b39f0 2024-03-29T11:45:42.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f12174d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:42.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f12174d0640 10 monclient(hunting): authenticate will time out at 1522.345087s 2024-03-29T11:45:42.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f121526c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f12100b4790 0x7f12100b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43078/0 (socket says 172.21.15.37:43078) 2024-03-29T11:45:42.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f121526c640 1 -- 172.21.15.37:0/3460941289 learned_addr learned my addr 172.21.15.37:0/3460941289 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:42.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f1207fff640 1 -- 172.21.15.37:0/3460941289 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705294257 0 0) 0x7f121013fbd0 con 0x7f12100b39f0 2024-03-29T11:45:42.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f1207fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:42.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f1207fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:42.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f1207fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:42.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f1207fff640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:45:42.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f1207fff640 1 -- 172.21.15.37:0/3460941289 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11fc003520 con 0x7f12100b39f0 2024-03-29T11:45:42.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f1207fff640 1 -- 172.21.15.37:0/3460941289 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789613343 0 0) 0x7f12100b0c60 con 0x7f12100b4790 2024-03-29T11:45:42.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f1207fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:42.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f1207fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f1207fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f1207fff640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.895+0000 7f1207fff640 1 -- 172.21.15.37:0/3460941289 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f121013fbd0 con 0x7f12100b4790 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 1 -- 172.21.15.37:0/3460941289 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2295566529 0 0) 0x7f11fc003520 con 0x7f12100b39f0 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 1 -- 172.21.15.37:0/3460941289 >> v1:172.21.15.146:6789/0 conn(0x7f12100aea90 legacy=0x7f12100af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 1 -- 172.21.15.37:0/3460941289 >> v1:172.21.15.37:6789/0 conn(0x7f12100b4790 legacy=0x7f12100b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 1 monclient: found mon.noname-b 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 1 -- 172.21.15.37:0/3460941289 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121013fd60 con 0x7f12100b39f0 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 10 monclient: _finish_auth 0 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:42.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f12174d0640 5 monclient: authenticate success, global_id 4670 2024-03-29T11:45:42.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f12174d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:42.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 1 -- 172.21.15.37:0/3460941289 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1208001690 con 0x7f12100b39f0 2024-03-29T11:45:42.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:42.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:42.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f1207fff640 10 monclient: dump: 2024-03-29T11:45:42.906 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:42.906 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:42.906 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:42.906 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:42.906 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:42.906 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:42.906 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:42.906 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.896+0000 7f12174d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.897+0000 7f1207fff640 1 -- 172.21.15.37:0/3460941289 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12080018b0 con 0x7f12100b39f0 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.897+0000 7f1207fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.897+0000 7f1207fff640 1 -- 172.21.15.37:0/3460941289 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1208001c40 con 0x7f12100b39f0 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.897+0000 7f12174d0640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.897+0000 7f12174d0640 4 set_mon_vals no callback set 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.897+0000 7f1207fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.897+0000 7f1207fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.897+0000 7f1207fff640 10 monclient: dump: 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:42.907 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.898+0000 7f12174d0640 10 monclient: shutdown 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.898+0000 7f12174d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.898+0000 7f12174d0640 1 -- 172.21.15.37:0/3460941289 >> v1:172.21.15.37:6790/0 conn(0x7f12100b39f0 legacy=0x7f121013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.898+0000 7f12174d0640 1 -- 172.21.15.37:0/3460941289 shutdown_connections 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.898+0000 7f12174d0640 1 -- 172.21.15.37:0/3460941289 >> 172.21.15.37:0/3460941289 conn(0x7f1210066a00 msgr2=0x7f1210066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.898+0000 7f12174d0640 1 -- 172.21.15.37:0/3460941289 shutdown_connections 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.898+0000 7f12174d0640 1 -- 172.21.15.37:0/3460941289 wait complete. 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.899+0000 7f12174d0640 10 monclient: build_initial_monmap 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.899+0000 7f12174d0640 10 monclient: monmap: 2024-03-29T11:45:42.908 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:42.909 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:42.909 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:45:42.909 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:45:42.909 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:42.909 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:42.909 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:42.909 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:45:42.909 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:45:42.909 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:42.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.899+0000 7f12174d0640 1 librados: starting msgr at 2024-03-29T11:45:42.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.899+0000 7f12174d0640 1 librados: starting objecter 2024-03-29T11:45:42.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.899+0000 7f12174d0640 1 Processor -- start 2024-03-29T11:45:42.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.899+0000 7f12174d0640 1 -- start start 2024-03-29T11:45:42.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.899+0000 7f12174d0640 1 librados: setting wanted keys 2024-03-29T11:45:42.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.899+0000 7f12174d0640 1 librados: calling monclient init 2024-03-29T11:45:42.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.899+0000 7f12174d0640 10 monclient: init 2024-03-29T11:45:42.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.900+0000 7f12174d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:42.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.900+0000 7f12174d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:42.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.900+0000 7f12174d0640 10 monclient(hunting): picked mon.noname-a con 0x7f12100b39f0 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:42.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.900+0000 7f12174d0640 10 monclient(hunting): picked mon.noname-c con 0x7f12100b4790 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:42.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.900+0000 7f12174d0640 10 monclient(hunting): picked mon.noname-b con 0x7f1210071490 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:42.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.900+0000 7f12174d0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12101e41a0 con 0x7f12100b39f0 2024-03-29T11:45:42.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.900+0000 7f12174d0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121006fa30 con 0x7f1210071490 2024-03-29T11:45:42.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.900+0000 7f12174d0640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12100a9e60 con 0x7f12100b4790 2024-03-29T11:45:42.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.900+0000 7f12174d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:42.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.900+0000 7f12174d0640 10 monclient(hunting): authenticate will time out at 1522.350372s 2024-03-29T11:45:42.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.900+0000 7f121526c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f12100b39f0 0x7f12100a9750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43304/0 (socket says 172.21.15.37:43304) 2024-03-29T11:45:42.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.900+0000 7f121526c640 1 -- 172.21.15.37:0/2836197743 learned_addr learned my addr 172.21.15.37:0/2836197743 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:42.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.900+0000 7f1205ffb640 10 client.?.objecter ms_handle_connect 0x7f12100b39f0 2024-03-29T11:45:42.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.900+0000 7f1205ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:42.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 10 client.?.objecter ms_handle_connect 0x7f12100b4790 2024-03-29T11:45:42.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:42.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33729495 0 0) 0x7f12100a9e60 con 0x7f12100b4790 2024-03-29T11:45:42.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:42.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:42.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:42.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 10 monclient(hunting): my global_id is 4673 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f4004eb0 con 0x7f12100b4790 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809286163 0 0) 0x7f12101e41a0 con 0x7f12100b39f0 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12100a9e60 con 0x7f12100b39f0 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 10 client.?.objecter ms_handle_connect 0x7f1210071490 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 429739528 0 0) 0x7f12100a9e60 con 0x7f12100b39f0 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12101e41a0 con 0x7f12100b39f0 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f120c001c90 con 0x7f12100b39f0 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.901+0000 7f1205ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:42.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 10 monclient(hunting): dump: 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818792774 0 0) 0x7f121006fa30 con 0x7f1210071490 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:42.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12100a9e60 con 0x7f1210071490 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3925760020 0 0) 0x7f11f4004eb0 con 0x7f12100b4790 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f121006fa30 con 0x7f12100b4790 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1200001510 con 0x7f12100b4790 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 10 monclient(hunting): dump: 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:42.917 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:42.918 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:42.918 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:42.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.902+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3118417332 0 0) 0x7f12101e41a0 con 0x7f12100b39f0 2024-03-29T11:45:42.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 >> v1:172.21.15.37:6790/0 conn(0x7f12100b4790 legacy=0x7f12101e46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:42.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 >> v1:172.21.15.146:6789/0 conn(0x7f1210071490 legacy=0x7f12101eebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:42.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f1205ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:45:42.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f1205ffb640 1 monclient: found mon.a 2024-03-29T11:45:42.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f1205ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:42.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f1205ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:42.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1210071fb0 con 0x7f12100b39f0 2024-03-29T11:45:42.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f1205ffb640 10 monclient: _finish_auth 0 2024-03-29T11:45:42.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f1205ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:45:42.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f1205ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:42.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f12174d0640 5 monclient: authenticate success, global_id 4599 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f12174d0640 10 monclient: _renew_subs 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f12174d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f12174d0640 1 -- 172.21.15.37:0/2836197743 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12101e4db0 con 0x7f12100b39f0 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f12174d0640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f12174d0640 10 monclient: _renew_subs 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f12174d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f12174d0640 1 -- 172.21.15.37:0/2836197743 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12101e5100 con 0x7f12100b39f0 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.903+0000 7f12174d0640 1 librados: init done 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.904+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f120c004b60 con 0x7f12100b39f0 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.904+0000 7f1205ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.904+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f120c004e70 con 0x7f12100b39f0 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.904+0000 7f1205ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.904+0000 7f1205ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.904+0000 7f1205ffb640 10 monclient: dump: 2024-03-29T11:45:42.919 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.904+0000 7f12077fe640 4 set_mon_vals no callback set 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.904+0000 7f12174d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.905+0000 7f12174d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.906+0000 7f12174d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.906+0000 7f12174d0640 1 -- 172.21.15.37:0/2836197743 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f0005080 con 0x7f12100b39f0 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.906+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f120c01ae40 con 0x7f12100b39f0 2024-03-29T11:45:42.920 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.907+0000 7f1214a6b640 10 monclient: get_auth_request con 0x7f11f406dfd0 auth_method 0 2024-03-29T11:45:42.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.907+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2864318804 0 0) 0x7f120c084dd0 con 0x7f12100b39f0 2024-03-29T11:45:42.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.907+0000 7f1205ffb640 10 client.4599.objecter ms_dispatch 0x7f1210000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:42.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.907+0000 7f1205ffb640 3 client.4599.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:42.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.907+0000 7f1205ffb640 3 client.4599.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:42.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.907+0000 7f1205ffb640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-29T11:45:42.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.907+0000 7f1205ffb640 10 client.4599.objecter ms_handle_connect 0x7f11f406dfd0 2024-03-29T11:45:42.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.916+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f120c0506b0 con 0x7f12100b39f0 2024-03-29T11:45:42.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.916+0000 7f1205ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:42.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:42.916+0000 7f1205ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:43.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.330+0000 7f12174d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:43.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.330+0000 7f12174d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:43.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.330+0000 7f12174d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:43.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.330+0000 7f12174d0640 1 -- 172.21.15.37:0/2836197743 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11f0005330 con 0x7f12100b39f0 2024-03-29T11:45:43.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.331+0000 7f1205ffb640 1 -- 172.21.15.37:0/2836197743 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1684400214) 0x7f120c0525e0 con 0x7f12100b39f0 2024-03-29T11:45:43.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.331+0000 7f1205ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:43.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.331+0000 7f1205ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:43.337 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:45:43.337 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:45:04.373638+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:04.373638+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:43.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.335+0000 7f12174d0640 10 librados: watch_flush enter 2024-03-29T11:45:43.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.335+0000 7f12174d0640 10 librados: watch_flush exit 2024-03-29T11:45:43.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.335+0000 7f12174d0640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-29T11:45:43.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.335+0000 7f12174d0640 10 client.4599.objecter successfully canceled tick 2024-03-29T11:45:43.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.335+0000 7f12174d0640 1 -- 172.21.15.37:0/2836197743 >> v1:172.21.15.146:6816/2899149139 conn(0x7f11f406dfd0 legacy=0x7f11f4070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:43.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.335+0000 7f12174d0640 10 monclient: shutdown 2024-03-29T11:45:43.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.335+0000 7f12174d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:43.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.335+0000 7f12174d0640 1 -- 172.21.15.37:0/2836197743 >> v1:172.21.15.37:6789/0 conn(0x7f12100b39f0 legacy=0x7f12100a9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:43.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.336+0000 7f12174d0640 1 -- 172.21.15.37:0/2836197743 shutdown_connections 2024-03-29T11:45:43.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.336+0000 7f12174d0640 1 -- 172.21.15.37:0/2836197743 >> 172.21.15.37:0/2836197743 conn(0x7f1210066a00 msgr2=0x7f12100b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:43.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.336+0000 7f12174d0640 1 -- 172.21.15.37:0/2836197743 shutdown_connections 2024-03-29T11:45:43.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.336+0000 7f12174d0640 1 -- 172.21.15.37:0/2836197743 wait complete. 2024-03-29T11:45:43.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:43.336+0000 7f12174d0640 1 librados: shutdown 2024-03-29T11:45:43.358 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:50.165 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:50.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.308+0000 7f5dac7f4640 10 monclient: get_monmap_and_config 2024-03-29T11:45:50.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.308+0000 7f5dac7f4640 10 monclient: build_initial_monmap 2024-03-29T11:45:50.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.308+0000 7f5dac7f4640 10 monclient: monmap: 2024-03-29T11:45:50.314 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:50.314 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:50.314 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:45:50.310741+0000 2024-03-29T11:45:50.314 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:45:50.310741+0000 2024-03-29T11:45:50.315 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:50.315 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:50.315 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:50.315 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:45:50.315 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:45:50.315 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:50.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.309+0000 7f5dac7f4640 1 Processor -- start 2024-03-29T11:45:50.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.309+0000 7f5dac7f4640 1 -- start start 2024-03-29T11:45:50.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.309+0000 7f5dac7f4640 10 monclient: init 2024-03-29T11:45:50.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.310+0000 7f5dac7f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:50.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.310+0000 7f5dac7f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:45:50.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.311+0000 7f5dac7f4640 10 monclient(hunting): picked mon.noname-c con 0x7f5da40708a0 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:50.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.311+0000 7f5dac7f4640 10 monclient(hunting): picked mon.noname-b con 0x7f5da406fe90 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:50.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.311+0000 7f5dac7f4640 10 monclient(hunting): picked mon.noname-a con 0x7f5da4143c50 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:50.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.311+0000 7f5dac7f4640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da4072630 con 0x7f5da4143c50 2024-03-29T11:45:50.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.311+0000 7f5dac7f4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da4071410 con 0x7f5da40708a0 2024-03-29T11:45:50.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.311+0000 7f5dac7f4640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da40afaa0 con 0x7f5da406fe90 2024-03-29T11:45:50.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.311+0000 7f5dac7f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:50.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.311+0000 7f5dac7f4640 10 monclient(hunting): authenticate will time out at 1529.761014s 2024-03-29T11:45:50.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.314+0000 7f5da9d8f640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f5da406fe90 0x7f5da4143540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:48313/0 (socket says 172.21.15.37:48313) 2024-03-29T11:45:50.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.314+0000 7f5da9d8f640 1 -- 172.21.15.37:0/892167539 learned_addr learned my addr 172.21.15.37:0/892167539 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:50.322 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.315+0000 7f5da958e640 1 -- 172.21.15.37:0/892167539 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347433790 0 0) 0x7f5da40afaa0 con 0x7f5da406fe90 2024-03-29T11:45:50.322 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.315+0000 7f5da958e640 10 monclient(hunting): _negotiate 2024-03-29T11:45:50.322 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.315+0000 7f5da958e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:50.322 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.315+0000 7f5da958e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:50.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.315+0000 7f5da958e640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:45:50.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.315+0000 7f5da958e640 1 -- 172.21.15.37:0/892167539 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d90003520 con 0x7f5da406fe90 2024-03-29T11:45:50.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.315+0000 7f5da958e640 1 -- 172.21.15.37:0/892167539 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3188623919 0 0) 0x7f5d90003520 con 0x7f5da406fe90 2024-03-29T11:45:50.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.315+0000 7f5da958e640 1 -- 172.21.15.37:0/892167539 >> v1:172.21.15.146:6789/0 conn(0x7f5da40708a0 legacy=0x7f5da40af390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:45:50.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.315+0000 7f5da958e640 1 -- 172.21.15.37:0/892167539 >> v1:172.21.15.37:6789/0 conn(0x7f5da4143c50 legacy=0x7f5da414b8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:50.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 10 monclient: _finish_hunting 0 2024-03-29T11:45:50.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 1 monclient: found mon.noname-b 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 1 -- 172.21.15.37:0/892167539 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da406dca0 con 0x7f5da406fe90 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 10 monclient: _finish_auth 0 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 10 monclient: _check_auth_tickets 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 1 -- 172.21.15.37:0/892167539 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5d9c001a10 con 0x7f5da406fe90 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 10 monclient: dump: 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:50.327 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5dac7f4640 5 monclient: authenticate success, global_id 4676 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5dac7f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 1 -- 172.21.15.37:0/892167539 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d9c001b70 con 0x7f5da406fe90 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 1 -- 172.21.15.37:0/892167539 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5d9c001fc0 con 0x7f5da406fe90 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5da958e640 10 monclient: dump: 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:50.328 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5dac7f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.318+0000 7f5dac7f4640 4 set_mon_vals no callback set 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.319+0000 7f5dac7f4640 10 monclient: shutdown 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.319+0000 7f5dac7f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.319+0000 7f5dac7f4640 1 -- 172.21.15.37:0/892167539 >> v1:172.21.15.37:6790/0 conn(0x7f5da406fe90 legacy=0x7f5da4143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.320+0000 7f5dac7f4640 1 -- 172.21.15.37:0/892167539 shutdown_connections 2024-03-29T11:45:50.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.320+0000 7f5dac7f4640 1 -- 172.21.15.37:0/892167539 >> 172.21.15.37:0/892167539 conn(0x7f5da40675b0 msgr2=0x7f5da40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:50.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.320+0000 7f5dac7f4640 1 -- 172.21.15.37:0/892167539 shutdown_connections 2024-03-29T11:45:50.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.321+0000 7f5dac7f4640 1 -- 172.21.15.37:0/892167539 wait complete. 2024-03-29T11:45:50.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.321+0000 7f5dac7f4640 10 monclient: build_initial_monmap 2024-03-29T11:45:50.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.321+0000 7f5dac7f4640 10 monclient: monmap: 2024-03-29T11:45:50.330 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:50.330 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:50.330 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:45:50.330 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:45:50.331 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:50.331 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:50.331 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:50.331 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:45:50.331 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:45:50.331 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:50.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.322+0000 7f5dac7f4640 1 librados: starting msgr at 2024-03-29T11:45:50.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.322+0000 7f5dac7f4640 1 librados: starting objecter 2024-03-29T11:45:50.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.322+0000 7f5dac7f4640 1 Processor -- start 2024-03-29T11:45:50.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.323+0000 7f5dac7f4640 1 -- start start 2024-03-29T11:45:50.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.323+0000 7f5dac7f4640 1 librados: setting wanted keys 2024-03-29T11:45:50.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.323+0000 7f5dac7f4640 1 librados: calling monclient init 2024-03-29T11:45:50.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.323+0000 7f5dac7f4640 10 monclient: init 2024-03-29T11:45:50.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.324+0000 7f5dac7f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:50.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.324+0000 7f5dac7f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:50.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.324+0000 7f5dac7f4640 10 monclient(hunting): picked mon.noname-b con 0x7f5da406fe90 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:50.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.324+0000 7f5dac7f4640 10 monclient(hunting): picked mon.noname-a con 0x7f5da40708a0 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:50.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.324+0000 7f5dac7f4640 10 monclient(hunting): picked mon.noname-c con 0x7f5da4143c50 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:50.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.324+0000 7f5dac7f4640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da40ab850 con 0x7f5da40708a0 2024-03-29T11:45:50.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.324+0000 7f5dac7f4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da4144e80 con 0x7f5da406fe90 2024-03-29T11:45:50.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.324+0000 7f5dac7f4640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da40abaf0 con 0x7f5da4143c50 2024-03-29T11:45:50.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.324+0000 7f5dac7f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:50.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.324+0000 7f5dac7f4640 10 monclient(hunting): authenticate will time out at 1529.774401s 2024-03-29T11:45:50.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.328+0000 7f5daa590640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5da406fe90 0x7f5da41e7580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:58323/0 (socket says 172.21.15.37:58323) 2024-03-29T11:45:50.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.328+0000 7f5daa590640 1 -- 172.21.15.37:0/3057941650 learned_addr learned my addr 172.21.15.37:0/3057941650 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:50.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.331+0000 7f5d9affd640 10 client.?.objecter ms_handle_connect 0x7f5da40708a0 2024-03-29T11:45:50.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.331+0000 7f5d9affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:50.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.331+0000 7f5d9affd640 10 client.?.objecter ms_handle_connect 0x7f5da4143c50 2024-03-29T11:45:50.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.331+0000 7f5d9affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:50.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.331+0000 7f5d9affd640 10 client.?.objecter ms_handle_connect 0x7f5da406fe90 2024-03-29T11:45:50.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.331+0000 7f5d9affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:50.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.331+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359040669 0 0) 0x7f5da40abaf0 con 0x7f5da4143c50 2024-03-29T11:45:50.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:50.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:50.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:50.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): my global_id is 4679 2024-03-29T11:45:50.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d88004eb0 con 0x7f5da4143c50 2024-03-29T11:45:50.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560438437 0 0) 0x7f5da4144e80 con 0x7f5da406fe90 2024-03-29T11:45:50.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:50.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:50.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:50.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:45:50.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da40abaf0 con 0x7f5da406fe90 2024-03-29T11:45:50.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3067585309 0 0) 0x7f5da40abaf0 con 0x7f5da406fe90 2024-03-29T11:45:50.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5da4144e80 con 0x7f5da406fe90 2024-03-29T11:45:50.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5da0001720 con 0x7f5da406fe90 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): dump: 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:50.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207879315 0 0) 0x7f5da40ab850 con 0x7f5da40708a0 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.332+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da40abaf0 con 0x7f5da40708a0 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2797105517 0 0) 0x7f5da4144e80 con 0x7f5da406fe90 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 >> v1:172.21.15.37:6790/0 conn(0x7f5da4143c50 legacy=0x7f5da41ef920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 >> v1:172.21.15.37:6789/0 conn(0x7f5da40708a0 legacy=0x7f5da4151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 1 monclient: found mon.b 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da41e7c90 con 0x7f5da406fe90 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 10 monclient: _finish_auth 0 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:50.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:50.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5da0001c30 con 0x7f5da406fe90 2024-03-29T11:45:50.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:50.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f5da0004aa0 con 0x7f5da406fe90 2024-03-29T11:45:50.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:50.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:50.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5d9affd640 10 monclient: dump: 2024-03-29T11:45:50.342 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:50.342 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:50.342 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:50.342 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:50.342 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:50.342 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:50.342 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:50.342 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:50.343 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:50.343 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:50.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.333+0000 7f5da8d8d640 4 set_mon_vals no callback set 2024-03-29T11:45:50.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.334+0000 7f5dac7f4640 5 monclient: authenticate success, global_id 4549 2024-03-29T11:45:50.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.334+0000 7f5dac7f4640 10 monclient: _renew_subs 2024-03-29T11:45:50.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.334+0000 7f5dac7f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:50.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.334+0000 7f5dac7f4640 1 -- 172.21.15.37:0/3057941650 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da40ac010 con 0x7f5da406fe90 2024-03-29T11:45:50.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.334+0000 7f5dac7f4640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:50.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.334+0000 7f5dac7f4640 10 monclient: _renew_subs 2024-03-29T11:45:50.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.334+0000 7f5dac7f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:50.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.334+0000 7f5dac7f4640 1 -- 172.21.15.37:0/3057941650 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da4140320 con 0x7f5da406fe90 2024-03-29T11:45:50.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.334+0000 7f5dac7f4640 1 librados: init done 2024-03-29T11:45:50.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.337+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f5da001aa70 con 0x7f5da406fe90 2024-03-29T11:45:50.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.338+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2864318804 0 0) 0x7f5da0084960 con 0x7f5da406fe90 2024-03-29T11:45:50.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.338+0000 7f5d9affd640 10 client.4549.objecter ms_dispatch 0x7f5da4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:50.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.338+0000 7f5d9affd640 3 client.4549.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:50.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.338+0000 7f5d9affd640 3 client.4549.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:50.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.339+0000 7f5d9affd640 20 client.4549.objecter dump_active .. 0 homeless 2024-03-29T11:45:50.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.339+0000 7f5dac7f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:50.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.339+0000 7f5da9d8f640 10 monclient: get_auth_request con 0x7f5d8806deb0 auth_method 0 2024-03-29T11:45:50.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.339+0000 7f5dac7f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:50.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.339+0000 7f5dac7f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:50.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.339+0000 7f5dac7f4640 1 -- 172.21.15.37:0/3057941650 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da4071e40 con 0x7f5da406fe90 2024-03-29T11:45:50.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.340+0000 7f5d9affd640 10 client.4549.objecter ms_handle_connect 0x7f5d8806deb0 2024-03-29T11:45:50.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.348+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5da0050240 con 0x7f5da406fe90 2024-03-29T11:45:50.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.348+0000 7f5d9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:50.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.348+0000 7f5d9affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:50.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.710+0000 7f5dac7f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:50.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.710+0000 7f5dac7f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:50.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.710+0000 7f5dac7f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-29T11:45:50.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.710+0000 7f5dac7f4640 1 -- 172.21.15.37:0/3057941650 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5da40acac0 con 0x7f5da406fe90 2024-03-29T11:45:50.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.712+0000 7f5d9affd640 1 -- 172.21.15.37:0/3057941650 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1684400214) 0x7f5da40acac0 con 0x7f5da406fe90 2024-03-29T11:45:50.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.712+0000 7f5d9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:50.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.712+0000 7f5d9affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:50.718 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:45:50.718 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:45:04.373638+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:04.373638+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:50.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.715+0000 7f5dac7f4640 10 librados: watch_flush enter 2024-03-29T11:45:50.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.715+0000 7f5dac7f4640 10 librados: watch_flush exit 2024-03-29T11:45:50.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.715+0000 7f5dac7f4640 20 client.4549.objecter shutdown clearing up homeless session... 2024-03-29T11:45:50.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.715+0000 7f5dac7f4640 10 client.4549.objecter successfully canceled tick 2024-03-29T11:45:50.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.715+0000 7f5dac7f4640 1 -- 172.21.15.37:0/3057941650 >> v1:172.21.15.146:6816/2899149139 conn(0x7f5d8806deb0 legacy=0x7f5d88070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:50.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.715+0000 7f5dac7f4640 10 monclient: shutdown 2024-03-29T11:45:50.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.715+0000 7f5dac7f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:50.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.715+0000 7f5dac7f4640 1 -- 172.21.15.37:0/3057941650 >> v1:172.21.15.146:6789/0 conn(0x7f5da406fe90 legacy=0x7f5da41e7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:50.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.715+0000 7f5dac7f4640 1 -- 172.21.15.37:0/3057941650 shutdown_connections 2024-03-29T11:45:50.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.715+0000 7f5dac7f4640 1 -- 172.21.15.37:0/3057941650 >> 172.21.15.37:0/3057941650 conn(0x7f5da40675b0 msgr2=0x7f5da40b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:50.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.716+0000 7f5dac7f4640 1 -- 172.21.15.37:0/3057941650 shutdown_connections 2024-03-29T11:45:50.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.716+0000 7f5dac7f4640 1 -- 172.21.15.37:0/3057941650 wait complete. 2024-03-29T11:45:50.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:50.716+0000 7f5dac7f4640 1 librados: shutdown 2024-03-29T11:45:50.736 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:57.542 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:57.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.691+0000 7f1caedcf640 10 monclient: get_monmap_and_config 2024-03-29T11:45:57.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.691+0000 7f1caedcf640 10 monclient: build_initial_monmap 2024-03-29T11:45:57.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.691+0000 7f1caedcf640 10 monclient: monmap: 2024-03-29T11:45:57.696 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:57.697 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:57.697 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:45:57.692376+0000 2024-03-29T11:45:57.697 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:45:57.692376+0000 2024-03-29T11:45:57.697 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:57.697 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:57.697 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:57.697 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-29T11:45:57.697 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-29T11:45:57.697 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:57.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.692+0000 7f1caedcf640 1 Processor -- start 2024-03-29T11:45:57.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.692+0000 7f1caedcf640 1 -- start start 2024-03-29T11:45:57.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.692+0000 7f1caedcf640 10 monclient: init 2024-03-29T11:45:57.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.693+0000 7f1caedcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:57.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.693+0000 7f1caedcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:57.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.693+0000 7f1caedcf640 10 monclient(hunting): picked mon.noname-a con 0x7f1ca806d9d0 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:57.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.693+0000 7f1caedcf640 10 monclient(hunting): picked mon.noname-b con 0x7f1ca813e820 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:57.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.693+0000 7f1caedcf640 10 monclient(hunting): picked mon.noname-c con 0x7f1ca80af1d0 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:57.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.693+0000 7f1caedcf640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca80aed10 con 0x7f1ca806d9d0 2024-03-29T11:45:57.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.693+0000 7f1caedcf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca80afd50 con 0x7f1ca80af1d0 2024-03-29T11:45:57.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.693+0000 7f1caedcf640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca80ab490 con 0x7f1ca813e820 2024-03-29T11:45:57.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.693+0000 7f1caedcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:57.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.693+0000 7f1caedcf640 10 monclient(hunting): authenticate will time out at 1537.142478s 2024-03-29T11:45:57.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.697+0000 7f1ca7fff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f1ca813e820 0x7f1ca813ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:37955/0 (socket says 172.21.15.37:37955) 2024-03-29T11:45:57.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.697+0000 7f1ca7fff640 1 -- 172.21.15.37:0/1769223737 learned_addr learned my addr 172.21.15.37:0/1769223737 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:57.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.697+0000 7f1ca77fe640 1 -- 172.21.15.37:0/1769223737 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895409301 0 0) 0x7f1ca80ab490 con 0x7f1ca813e820 2024-03-29T11:45:57.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.697+0000 7f1ca77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:57.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.697+0000 7f1ca77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:57.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.697+0000 7f1ca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:57.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.698+0000 7f1ca77fe640 10 monclient(hunting): my global_id is 4682 2024-03-29T11:45:57.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.698+0000 7f1ca77fe640 1 -- 172.21.15.37:0/1769223737 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c90003520 con 0x7f1ca813e820 2024-03-29T11:45:57.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.699+0000 7f1ca77fe640 1 -- 172.21.15.37:0/1769223737 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551367259 0 0) 0x7f1ca80afd50 con 0x7f1ca80af1d0 2024-03-29T11:45:57.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.699+0000 7f1ca77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:57.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.699+0000 7f1ca77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:57.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.699+0000 7f1ca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:57.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.699+0000 7f1ca77fe640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:45:57.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.699+0000 7f1ca77fe640 1 -- 172.21.15.37:0/1769223737 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ca80ab490 con 0x7f1ca80af1d0 2024-03-29T11:45:57.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.699+0000 7f1ca77fe640 1 -- 172.21.15.37:0/1769223737 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 333276257 0 0) 0x7f1c90003520 con 0x7f1ca813e820 2024-03-29T11:45:57.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.699+0000 7f1ca77fe640 1 -- 172.21.15.37:0/1769223737 >> v1:172.21.15.146:6789/0 conn(0x7f1ca80af1d0 legacy=0x7f1ca80af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:57.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.699+0000 7f1ca77fe640 1 -- 172.21.15.37:0/1769223737 >> v1:172.21.15.37:6789/0 conn(0x7f1ca806d9d0 legacy=0x7f1ca80b05f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:57.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1ca77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:45:57.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1ca77fe640 1 monclient: found mon.noname-b 2024-03-29T11:45:57.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1ca77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:57.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1ca77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-29T11:45:57.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1ca77fe640 1 -- 172.21.15.37:0/1769223737 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca80ab770 con 0x7f1ca813e820 2024-03-29T11:45:57.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1ca77fe640 10 monclient: _finish_auth 0 2024-03-29T11:45:57.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1ca77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1ca77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1caedcf640 5 monclient: authenticate success, global_id 4682 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1caedcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1ca77fe640 1 -- 172.21.15.37:0/1769223737 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1c9c001690 con 0x7f1ca813e820 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1ca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1ca77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1ca77fe640 10 monclient: dump: 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:57.706 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:57.707 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:57.707 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:57.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.700+0000 7f1caedcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:57.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.703+0000 7f1ca77fe640 1 -- 172.21.15.37:0/1769223737 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c9c0017f0 con 0x7f1ca813e820 2024-03-29T11:45:57.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.703+0000 7f1ca77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:57.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.703+0000 7f1caedcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:57.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.703+0000 7f1caedcf640 4 set_mon_vals no callback set 2024-03-29T11:45:57.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.704+0000 7f1ca77fe640 1 -- 172.21.15.37:0/1769223737 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1c9c001c00 con 0x7f1ca813e820 2024-03-29T11:45:57.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.704+0000 7f1ca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:57.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.704+0000 7f1ca77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:57.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.704+0000 7f1ca77fe640 10 monclient: dump: 2024-03-29T11:45:57.709 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:57.709 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:57.709 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:57.709 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:57.710 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:57.710 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:57.710 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:57.710 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:57.710 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:57.710 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:57.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.704+0000 7f1caedcf640 10 monclient: shutdown 2024-03-29T11:45:57.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.704+0000 7f1caedcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:57.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.704+0000 7f1caedcf640 1 -- 172.21.15.37:0/1769223737 >> v1:172.21.15.37:6790/0 conn(0x7f1ca813e820 legacy=0x7f1ca813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:57.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.704+0000 7f1caedcf640 1 -- 172.21.15.37:0/1769223737 shutdown_connections 2024-03-29T11:45:57.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.704+0000 7f1caedcf640 1 -- 172.21.15.37:0/1769223737 >> 172.21.15.37:0/1769223737 conn(0x7f1ca806b660 msgr2=0x7f1ca806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:57.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.705+0000 7f1caedcf640 1 -- 172.21.15.37:0/1769223737 shutdown_connections 2024-03-29T11:45:57.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.705+0000 7f1caedcf640 1 -- 172.21.15.37:0/1769223737 wait complete. 2024-03-29T11:45:57.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.706+0000 7f1caedcf640 10 monclient: build_initial_monmap 2024-03-29T11:45:57.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.706+0000 7f1caedcf640 10 monclient: monmap: 2024-03-29T11:45:57.712 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-29T11:45:57.712 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:57.712 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-29T11:45:57.712 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-29T11:45:57.712 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:57.712 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:57.712 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-29T11:45:57.712 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-29T11:45:57.712 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-29T11:45:57.712 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:57.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.706+0000 7f1caedcf640 1 librados: starting msgr at 2024-03-29T11:45:57.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.706+0000 7f1caedcf640 1 librados: starting objecter 2024-03-29T11:45:57.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.707+0000 7f1caedcf640 1 Processor -- start 2024-03-29T11:45:57.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.707+0000 7f1caedcf640 1 -- start start 2024-03-29T11:45:57.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.707+0000 7f1caedcf640 1 librados: setting wanted keys 2024-03-29T11:45:57.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.707+0000 7f1caedcf640 1 librados: calling monclient init 2024-03-29T11:45:57.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.707+0000 7f1caedcf640 10 monclient: init 2024-03-29T11:45:57.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.708+0000 7f1caedcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:57.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.708+0000 7f1caedcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:57.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.708+0000 7f1caedcf640 10 monclient(hunting): picked mon.noname-c con 0x7f1ca806d9d0 addr v1:172.21.15.37:6790/0 2024-03-29T11:45:57.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.708+0000 7f1caedcf640 10 monclient(hunting): picked mon.noname-a con 0x7f1ca80af1d0 addr v1:172.21.15.37:6789/0 2024-03-29T11:45:57.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.708+0000 7f1caedcf640 10 monclient(hunting): picked mon.noname-b con 0x7f1ca806e050 addr v1:172.21.15.146:6789/0 2024-03-29T11:45:57.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.708+0000 7f1caedcf640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca806f360 con 0x7f1ca80af1d0 2024-03-29T11:45:57.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.708+0000 7f1caedcf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca812b060 con 0x7f1ca806e050 2024-03-29T11:45:57.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.708+0000 7f1caedcf640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca80b28f0 con 0x7f1ca806d9d0 2024-03-29T11:45:57.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.708+0000 7f1caedcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:57.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.708+0000 7f1caedcf640 10 monclient(hunting): authenticate will time out at 1537.157447s 2024-03-29T11:45:57.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.712+0000 7f1cad36c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1ca806e050 0x7f1ca81eef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.37:41539/0 (socket says 172.21.15.37:41539) 2024-03-29T11:45:57.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.713+0000 7f1cad36c640 1 -- 172.21.15.37:0/2689747361 learned_addr learned my addr 172.21.15.37:0/2689747361 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-29T11:45:57.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.714+0000 7f1ca57fa640 10 client.?.objecter ms_handle_connect 0x7f1ca806e050 2024-03-29T11:45:57.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.714+0000 7f1ca57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:57.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.714+0000 7f1ca57fa640 10 client.?.objecter ms_handle_connect 0x7f1ca80af1d0 2024-03-29T11:45:57.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.714+0000 7f1ca57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:57.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.714+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257539396 0 0) 0x7f1ca806f360 con 0x7f1ca80af1d0 2024-03-29T11:45:57.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.714+0000 7f1ca57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:57.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.714+0000 7f1ca57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:57.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.714+0000 7f1ca57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:57.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.714+0000 7f1ca57fa640 10 monclient(hunting): my global_id is 4605 2024-03-29T11:45:57.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.714+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c94004eb0 con 0x7f1ca80af1d0 2024-03-29T11:45:57.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.717+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4074801594 0 0) 0x7f1c94004eb0 con 0x7f1ca80af1d0 2024-03-29T11:45:57.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.717+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ca806f360 con 0x7f1ca80af1d0 2024-03-29T11:45:57.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.717+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654253859 0 0) 0x7f1ca812b060 con 0x7f1ca806e050 2024-03-29T11:45:57.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.717+0000 7f1ca57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:57.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.717+0000 7f1ca57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.717+0000 7f1ca57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.717+0000 7f1ca57fa640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.717+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c94004eb0 con 0x7f1ca806e050 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.717+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1c9c001b00 con 0x7f1ca80af1d0 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.717+0000 7f1ca57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.717+0000 7f1ca57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.717+0000 7f1ca57fa640 10 monclient(hunting): dump: 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:57.725 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.717+0000 7f1ca57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.717+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2067041498 0 0) 0x7f1ca806f360 con 0x7f1ca80af1d0 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 >> v1:172.21.15.37:6790/0 conn(0x7f1ca806d9d0 legacy=0x7f1ca8134d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 >> v1:172.21.15.146:6789/0 conn(0x7f1ca806e050 legacy=0x7f1ca81eef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1ca57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1ca57fa640 1 monclient: found mon.a 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1ca57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1ca57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca80b4370 con 0x7f1ca80af1d0 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1ca57fa640 10 monclient: _finish_auth 0 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1ca57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1ca57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:57.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1caedcf640 5 monclient: authenticate success, global_id 4605 2024-03-29T11:45:57.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1caedcf640 10 monclient: _renew_subs 2024-03-29T11:45:57.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1caedcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:57.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1caedcf640 1 -- 172.21.15.37:0/2689747361 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca806f780 con 0x7f1ca80af1d0 2024-03-29T11:45:57.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1caedcf640 10 client.4605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:57.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1caedcf640 10 monclient: _renew_subs 2024-03-29T11:45:57.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1caedcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:57.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1caedcf640 1 -- 172.21.15.37:0/2689747361 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca814c390 con 0x7f1ca80af1d0 2024-03-29T11:45:57.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.718+0000 7f1caedcf640 1 librados: init done 2024-03-29T11:45:57.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.722+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c9c004a90 con 0x7f1ca80af1d0 2024-03-29T11:45:57.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.722+0000 7f1ca57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:57.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.722+0000 7f1ca6ffd640 4 set_mon_vals no callback set 2024-03-29T11:45:57.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.722+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3665911314 0 0) 0x7f1c9c004da0 con 0x7f1ca80af1d0 2024-03-29T11:45:57.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.722+0000 7f1ca57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:57.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.722+0000 7f1ca57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:57.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.722+0000 7f1ca57fa640 10 monclient: dump: 2024-03-29T11:45:57.729 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-29T11:45:57.729 INFO:teuthology.misc.health.smithi037.stderr:fsid 5a9df19a-9269-4a6f-b895-975ff91e9fb9 2024-03-29T11:45:57.729 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:57.729 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-29T11:34:21.294342+0000 2024-03-29T11:45:57.729 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:57.729 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-29T11:45:57.729 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-29T11:45:57.729 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-29T11:45:57.729 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-29T11:45:57.729 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-29T11:45:57.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.724+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 485803805 0 0) 0x7f1c9c01ad70 con 0x7f1ca80af1d0 2024-03-29T11:45:57.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.724+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2864318804 0 0) 0x7f1c9c054720 con 0x7f1ca80af1d0 2024-03-29T11:45:57.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.724+0000 7f1ca57fa640 10 client.4605.objecter ms_dispatch 0x7f1ca8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:57.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.724+0000 7f1ca57fa640 3 client.4605.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:57.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.724+0000 7f1ca57fa640 3 client.4605.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:57.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.724+0000 7f1ca57fa640 20 client.4605.objecter dump_active .. 0 homeless 2024-03-29T11:45:57.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.724+0000 7f1caedcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:57.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.725+0000 7f1caedcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:57.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.725+0000 7f1caedcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:57.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.725+0000 7f1caedcf640 1 -- 172.21.15.37:0/2689747361 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c88005080 con 0x7f1ca80af1d0 2024-03-29T11:45:57.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.725+0000 7f1cacb6b640 10 monclient: get_auth_request con 0x7f1c9406de80 auth_method 0 2024-03-29T11:45:57.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.726+0000 7f1ca57fa640 10 client.4605.objecter ms_handle_connect 0x7f1c9406de80 2024-03-29T11:45:57.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.735+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c9c0509c0 con 0x7f1ca80af1d0 2024-03-29T11:45:57.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.736+0000 7f1ca57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:57.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:57.736+0000 7f1ca57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:58.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.068+0000 7f1caedcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:58.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.068+0000 7f1caedcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:58.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.069+0000 7f1caedcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-29T11:45:58.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.069+0000 7f1caedcf640 1 -- 172.21.15.37:0/2689747361 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c88005330 con 0x7f1ca80af1d0 2024-03-29T11:45:58.076 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.071+0000 7f1ca57fa640 1 -- 172.21.15.37:0/2689747361 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1684400214) 0x7f1c9c0509c0 con 0x7f1ca80af1d0 2024-03-29T11:45:58.076 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.071+0000 7f1ca57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:58.076 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.071+0000 7f1ca57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:58.076 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-29T11:45:58.077 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"5a9df19a-9269-4a6f-b895-975ff91e9fb9","created":"2024-03-29T11:34:59.435148+0000","modified":"2024-03-29T11:45:04.373638+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:04.373638+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c8cbb56-9953-4797-b700-981d68eda10b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9a7adcae-7e22-4730-9149-d05cd77f3034","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9d447705-705c-4657-ba3d-5ab095b713a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b73b3c8f-d12d-4c7e-8df9-637a89f5df27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c2d3d7e-d6a6-4748-b9bf-4ac4f6b084ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8adc2525-35e0-477a-8946-2a6afe773016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"634a1b82-a618-42c8-b306-b42760f2e055","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35f0fe13-712e-4137-b072-6f0db1b9789b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:58.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.075+0000 7f1caedcf640 10 librados: watch_flush enter 2024-03-29T11:45:58.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.075+0000 7f1caedcf640 10 librados: watch_flush exit 2024-03-29T11:45:58.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.075+0000 7f1caedcf640 20 client.4605.objecter shutdown clearing up homeless session... 2024-03-29T11:45:58.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.075+0000 7f1caedcf640 10 client.4605.objecter successfully canceled tick 2024-03-29T11:45:58.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.075+0000 7f1caedcf640 1 -- 172.21.15.37:0/2689747361 >> v1:172.21.15.146:6816/2899149139 conn(0x7f1c9406de80 legacy=0x7f1c94070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:58.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.075+0000 7f1caedcf640 10 monclient: shutdown 2024-03-29T11:45:58.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.075+0000 7f1caedcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:58.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.075+0000 7f1caedcf640 1 -- 172.21.15.37:0/2689747361 >> v1:172.21.15.37:6789/0 conn(0x7f1ca80af1d0 legacy=0x7f1ca80b21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:58.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.076+0000 7f1caedcf640 1 -- 172.21.15.37:0/2689747361 shutdown_connections 2024-03-29T11:45:58.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.076+0000 7f1caedcf640 1 -- 172.21.15.37:0/2689747361 >> 172.21.15.37:0/2689747361 conn(0x7f1ca806b660 msgr2=0x7f1ca814d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:58.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.076+0000 7f1caedcf640 1 -- 172.21.15.37:0/2689747361 shutdown_connections 2024-03-29T11:45:58.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.076+0000 7f1caedcf640 1 -- 172.21.15.37:0/2689747361 wait complete. 2024-03-29T11:45:58.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-29T11:45:58.076+0000 7f1caedcf640 1 librados: shutdown 2024-03-29T11:45:58.097 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:58.098 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:45:58.099 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:45:58.100 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:45:58.100 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.136 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:45:58.136 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:45:58.137 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.214 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:45:58.214 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:45:58.214 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.289 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:45:58.289 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:45:58.289 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.363 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:45:58.363 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:45:58.363 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.397 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:45:58.397 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:45:58.397 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.471 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:45:58.472 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:45:58.472 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.549 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:45:58.550 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:45:58.550 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.617 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:45:58.617 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:45:58.618 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:45:58.618 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.641 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:45:58.642 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:45:58.642 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.685 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:45:58.686 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:45:58.686 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:45:58.686 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.697 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:45:58.697 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:45:58.697 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.749 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:45:58.749 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:45:58.749 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.760 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:45:58.760 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:45:58.760 DEBUG:teuthology.orchestra.run.smithi037:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:45:58.826 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-29T11:45:58.827 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:46:00.222 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-29T11:46:00.222 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:46:00.312 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-29T11:46:00.312 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:46:00.353 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-29T11:46:00.354 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:46:00.453 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-29T11:46:00.454 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:46:01.877 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-29T11:46:01.877 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:46:01.954 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-29T11:46:01.954 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:46:01.994 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-29T11:46:01.994 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:46:02.093 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:46:02.095 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721/data/mon.a.tgz 2024-03-29T11:46:02.095 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-03-29T11:46:02.114 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.KBaGOJm4yQ 2024-03-29T11:46:02.115 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.KBaGOJm4yQ 2024-03-29T11:46:02.214 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.KBaGOJm4yQ 2024-03-29T11:46:02.302 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.KBaGOJm4yQ is 126KB 2024-03-29T11:46:02.353 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.KBaGOJm4yQ 2024-03-29T11:46:02.368 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721/data/mon.c.tgz 2024-03-29T11:46:02.368 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-03-29T11:46:02.422 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.pgsPJLhKQZ 2024-03-29T11:46:02.422 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.pgsPJLhKQZ 2024-03-29T11:46:02.511 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.pgsPJLhKQZ 2024-03-29T11:46:02.595 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.pgsPJLhKQZ is 126KB 2024-03-29T11:46:02.645 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.pgsPJLhKQZ 2024-03-29T11:46:02.658 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721/data/mon.b.tgz 2024-03-29T11:46:02.659 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-03-29T11:46:02.676 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.iDfIn3SDcq 2024-03-29T11:46:02.676 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.iDfIn3SDcq 2024-03-29T11:46:02.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.iDfIn3SDcq 2024-03-29T11:46:02.865 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.iDfIn3SDcq is 126KB 2024-03-29T11:46:02.916 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.iDfIn3SDcq 2024-03-29T11:46:02.931 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:46:02.931 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:46:02.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:46:03.016 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:46:03.056 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:46:03.056 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:46:03.058 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721/remote/smithi037/crash 2024-03-29T11:46:03.059 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:46:03.085 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721/remote/smithi146/crash 2024-03-29T11:46:03.086 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:46:03.122 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:46:03.122 DEBUG:teuthology.orchestra.run.smithi037:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:46:03.129 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:46:03.198 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41082.log 2024-03-29T11:46:03.198 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38852.log 2024-03-29T11:46:03.198 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41082.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:46:03.198 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38109.log 2024-03-29T11:46:03.199 INFO:teuthology.orchestra.run.smithi037.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41082.log.gz 2024-03-29T11:46:03.199 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39676.log 2024-03-29T11:46:03.199 INFO:teuthology.orchestra.run.smithi037.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38852.log.gz 2024-03-29T11:46:03.199 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38031.log 2024-03-29T11:46:03.199 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38109.log: /var/log/ceph/ceph-client.admin.39676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40652.log 2024-03-29T11:46:03.199 INFO:teuthology.orchestra.run.smithi037.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38109.log.gz 2024-03-29T11:46:03.199 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40379.log 2024-03-29T11:46:03.199 INFO:teuthology.orchestra.run.smithi037.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39676.log.gz 2024-03-29T11:46:03.199 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40223.log 2024-03-29T11:46:03.199 INFO:teuthology.orchestra.run.smithi037.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38031.log.gz 2024-03-29T11:46:03.199 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39598.log 2024-03-29T11:46:03.199 INFO:teuthology.orchestra.run.smithi037.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40652.log.gz 2024-03-29T11:46:03.199 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40340.log 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40379.log.gz 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39598.log: gzip 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40223.log.gz -5 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37875.log 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41043.log 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39598.log.gz 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39242.log 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40340.log.gz 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40496.log 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37875.log.gz 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38423.log 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41043.log.gz 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38384.log 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39242.log.gz 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38423.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40496.log.gz 2024-03-29T11:46:03.200 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38267.log 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38501.log 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38384.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38423.log.gz/var/log/ceph/ceph-client.admin.38267.log: 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40613.log 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38384.log.gz 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38070.log 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38267.log.gz 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40692.log 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40613.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38501.log.gz 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40887.log 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40613.log.gz 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39517.log 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38070.log.gz 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41238.log 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40692.log.gz 2024-03-29T11:46:03.201 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39517.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40887.log.gz 2024-03-29T11:46:03.202 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40106.log 2024-03-29T11:46:03.202 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38306.log 2024-03-29T11:46:03.202 INFO:teuthology.orchestra.run.smithi037.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39517.log.gz 2024-03-29T11:46:03.202 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40067.log 2024-03-29T11:46:03.202 INFO:teuthology.orchestra.run.smithi037.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41238.log.gz 2024-03-29T11:46:03.202 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39476.log 2024-03-29T11:46:03.202 INFO:teuthology.orchestra.run.smithi037.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40106.log.gz 2024-03-29T11:46:03.202 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40067.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38306.log.gz 2024-03-29T11:46:03.202 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40145.log 2024-03-29T11:46:03.202 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40731.log 2024-03-29T11:46:03.202 INFO:teuthology.orchestra.run.smithi037.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40067.log.gz 2024-03-29T11:46:03.202 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38579.log 2024-03-29T11:46:03.202 INFO:teuthology.orchestra.run.smithi037.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39476.log.gz 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40731.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40145.log.gz 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40731.log.gz 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40926.log 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39008.log 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41277.log 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38579.log.gz 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37914.log 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40926.log.gz 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41277.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39008.log.gz 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40028.log 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41277.log.gz 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41199.log 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37914.log.gz 2024-03-29T11:46:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40535.log 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41199.log: 93.4% 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40028.log.gz -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38189.log 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39716.log 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41199.log.gz 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40770.log 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40535.log.gz 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40301.log 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38189.log.gz 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40809.log 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39716.log.gz 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41121.log 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40770.log.gz 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40301.log.gz 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40809.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:46:03.204 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39164.log 2024-03-29T11:46:03.205 INFO:teuthology.orchestra.run.smithi037.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40809.log.gz 2024-03-29T11:46:03.205 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose 86.4% -- -- replaced with /var/log/ceph/ceph-client.admin.41121.log.gz /var/log/ceph/ceph-client.admin.40574.log 2024-03-29T11:46:03.205 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:46:03.205 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39637.log 2024-03-29T11:46:03.205 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39911.log 2024-03-29T11:46:03.205 INFO:teuthology.orchestra.run.smithi037.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39164.log.gz 2024-03-29T11:46:03.205 INFO:teuthology.orchestra.run.smithi037.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40574.log.gz 2024-03-29T11:46:03.205 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39833.log 2024-03-29T11:46:03.205 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39755.log 2024-03-29T11:46:03.205 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39833.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39637.log.gz 2024-03-29T11:46:03.205 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40184.log 2024-03-29T11:46:03.205 INFO:teuthology.orchestra.run.smithi037.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39911.log.gz 2024-03-29T11:46:03.205 INFO:teuthology.orchestra.run.smithi037.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39833.log.gz 2024-03-29T11:46:03.205 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39755.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:46:03.206 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38891.log 2024-03-29T11:46:03.206 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.3.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39755.log.gz 2024-03-29T11:46:03.206 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38774.log 2024-03-29T11:46:03.206 INFO:teuthology.orchestra.run.smithi037.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40184.log.gz 2024-03-29T11:46:03.206 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5/var/log/ceph/ceph-client.admin.38891.log: --verbose -- /var/log/ceph/ceph-client.admin.38813.log 2024-03-29T11:46:03.206 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40418.log 2024-03-29T11:46:03.206 INFO:teuthology.orchestra.run.smithi037.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38891.log.gz 2024-03-29T11:46:03.206 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38813.log: gzip 85.3% -- replaced with /var/log/ceph/ceph-client.admin.38774.log.gz 2024-03-29T11:46:03.206 INFO:teuthology.orchestra.run.smithi037.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38657.log 2024-03-29T11:46:03.206 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40418.log: gzip -5 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38813.log.gz 2024-03-29T11:46:03.206 INFO:teuthology.orchestra.run.smithi037.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39203.log 2024-03-29T11:46:03.206 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38657.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40418.log.gz 2024-03-29T11:46:03.206 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38930.log 2024-03-29T11:46:03.206 INFO:teuthology.orchestra.run.smithi037.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38657.log.gz 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39047.log 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38930.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39203.log.gz 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41160.log 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38930.log.gz 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39950.log 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38618.log 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39047.log.gz 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39950.log: gzip -5 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41160.log.gz 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr: --verbose -- /var/log/ceph/ceph.tmp-client.admin.28460.log 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38618.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39950.log.gz 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39398.log 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38618.log.gz 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph.tmp-client.admin.28460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34570.log 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28460.log.gz 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39398.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34570.log: gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39398.log.gz 2024-03-29T11:46:03.207 INFO:teuthology.orchestra.run.smithi037.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38969.log 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-mgr.y.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34570.log.gz 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38540.log 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39320.log 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38540.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38969.log.gz 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41004.log 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39320.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38540.log.gz 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39794.log 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41004.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39320.log.gz 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41316.log 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41004.log.gz 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40965.log 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41316.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39794.log.gz 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40457.log 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40965.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41316.log.gz 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:46:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40457.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40965.log.gz 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38462.log 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.0.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40457.log.gz 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39989.log 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38462.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39989.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38462.log.gz 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39556.log 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39989.log.gz 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39359.log 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39556.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39086.log 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39359.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39556.log.gz 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39359.log.gz 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39437.log 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39086.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39086.log.gz 2024-03-29T11:46:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37953.log 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34673.log 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37953.log: gzip -5 --verbose -- 87.0% /var/log/ceph/ceph-client.admin.39872.log 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39437.log.gz 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.37953.log.gz 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34673.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34673.log.gz 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40848.log 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38696.log 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40848.log: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38150.log 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39872.log.gz 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40848.log.gz 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38696.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38696.log.gz 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34279.log 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38735.log 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34279.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38150.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39125.log 2024-03-29T11:46:03.210 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:46:03.211 INFO:teuthology.orchestra.run.smithi037.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34279.log.gz 2024-03-29T11:46:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38735.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38735.log.gz 2024-03-29T11:46:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38345.log 2024-03-29T11:46:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37992.log 2024-03-29T11:46:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38345.log: 86.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39281.log 2024-03-29T11:46:03.211 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39125.log.gz 2024-03-29T11:46:03.211 INFO:teuthology.orchestra.run.smithi037.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38345.log.gz/var/log/ceph/ceph-client.admin.37992.log: gzip 2024-03-29T11:46:03.211 INFO:teuthology.orchestra.run.smithi037.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40262.log 2024-03-29T11:46:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39281.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:46:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40262.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.37992.log.gz 2024-03-29T11:46:03.211 INFO:teuthology.orchestra.run.smithi037.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39281.log.gz 2024-03-29T11:46:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38228.log 2024-03-29T11:46:03.211 INFO:teuthology.orchestra.run.smithi037.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40262.log.gz 2024-03-29T11:46:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.38228.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38228.log.gz 2024-03-29T11:46:03.214 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34423.log 2024-03-29T11:46:03.214 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:46:03.214 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.34423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34383.log 2024-03-29T11:46:03.214 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34263.log 2024-03-29T11:46:03.214 INFO:teuthology.orchestra.run.smithi146.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34423.log.gz 2024-03-29T11:46:03.214 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.34383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34303.log 2024-03-29T11:46:03.214 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.34263.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.34383.log.gz 2024-03-29T11:46:03.214 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:46:03.214 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.34303.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34263.log.gz 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34303.log.gz 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34143.log 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34223.log 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.34143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34183.log 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.34223.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34143.log.gz 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.34183.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34223.log.gz 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34183.log.gz 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34343.log 2024-03-29T11:46:03.215 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.34343.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34343.log.gz 2024-03-29T11:46:03.265 INFO:teuthology.orchestra.run.smithi146.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:46:15.060 INFO:teuthology.orchestra.run.smithi146.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:46:15.087 INFO:teuthology.orchestra.run.smithi146.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:46:15.130 INFO:teuthology.orchestra.run.smithi146.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:46:15.226 INFO:teuthology.orchestra.run.smithi037.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:46:19.147 INFO:teuthology.orchestra.run.smithi146.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:46:19.366 INFO:teuthology.orchestra.run.smithi037.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:46:19.401 INFO:teuthology.orchestra.run.smithi037.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:46:21.003 INFO:teuthology.orchestra.run.smithi037.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:46:51.278 INFO:teuthology.orchestra.run.smithi037.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:47:00.390 INFO:teuthology.orchestra.run.smithi037.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:47:00.391 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-29T11:47:00.391 INFO:teuthology.orchestra.run.smithi037.stderr:real 0m57.249s 2024-03-29T11:47:00.391 INFO:teuthology.orchestra.run.smithi037.stderr:user 2m42.012s 2024-03-29T11:47:00.392 INFO:teuthology.orchestra.run.smithi037.stderr:sys 0m5.264s 2024-03-29T11:47:13.684 INFO:teuthology.orchestra.run.smithi146.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:47:13.685 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-29T11:47:13.685 INFO:teuthology.orchestra.run.smithi146.stderr:real 1m10.504s 2024-03-29T11:47:13.685 INFO:teuthology.orchestra.run.smithi146.stderr:user 1m57.515s 2024-03-29T11:47:13.686 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m4.485s 2024-03-29T11:47:13.686 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:47:13.686 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721/remote/smithi037/log 2024-03-29T11:47:13.687 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:47:36.959 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721/remote/smithi146/log 2024-03-29T11:47:36.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:47:55.508 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:47:55.719 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=98164ac551f14d6386e6b1b3fc2f6a20 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:47:55.723 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:47:55.731 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:47:55.738 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:47:55.739 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:47:55.753 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:47:55.832 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:47:55.841 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:47:55.841 DEBUG:teuthology.orchestra.run.smithi037:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:47:55.844 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:47:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:47:55.860 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:47:55.862 INFO:teuthology.orchestra.run.smithi037.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:47:55.862 INFO:teuthology.orchestra.run.smithi037.stdout:=============================================================================== 2024-03-29T11:47:55.862 INFO:teuthology.orchestra.run.smithi037.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 +390us[ +388us] +/- 73ms 2024-03-29T11:47:55.862 INFO:teuthology.orchestra.run.smithi037.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -123us[ -125us] +/- 68ms 2024-03-29T11:47:55.862 INFO:teuthology.orchestra.run.smithi037.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -234us[ -237us] +/- 68ms 2024-03-29T11:47:55.862 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:47:55.865 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:47:55.865 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-03-29T11:47:55.865 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 35 +401us[ +401us] +/- 73ms 2024-03-29T11:47:55.865 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -126us[ -134us] +/- 68ms 2024-03-29T11:47:55.865 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 -227us[ -227us] +/- 68ms 2024-03-29T11:47:55.865 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:47:55.866 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:47:55.875 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:47:55.876 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:47:55.884 DEBUG:teuthology.orchestra.run.smithi037:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:47:55.975 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:47:56.048 DEBUG:teuthology.orchestra.run.smithi037:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:47:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:56.080 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:47:56.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:56.112 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:47:56.122 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:47:56.130 INFO:teuthology.task.internal:Duration was 1279.809854 seconds 2024-03-29T11:47:56.131 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:47:56.139 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:47:56.139 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:47:56.142 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:47:56.184 INFO:teuthology.orchestra.run.smithi037.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:47:56.197 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:47:56.583 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:47:56.584 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi037.front.sepia.ceph.com 2024-03-29T11:47:56.584 DEBUG:teuthology.orchestra.run.smithi037:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:47:56.645 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-03-29T11:47:56.645 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:47:56.667 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:47:56.668 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:47:56.688 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:47:56.737 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:47:56.737 DEBUG:teuthology.orchestra.run.smithi037:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:47:56.760 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:47:56.874 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:47:56.883 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:47:56.884 DEBUG:teuthology.orchestra.run.smithi037:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:47:56.912 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:47:56.939 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:47:56.949 DEBUG:teuthology.orchestra.run.smithi037:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:56.953 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:56.977 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = core 2024-03-29T11:47:57.004 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-03-29T11:47:57.055 DEBUG:teuthology.orchestra.run.smithi037:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:57.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:57.069 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:57.109 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:47:57.123 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:47:57.125 DEBUG:teuthology.misc:Transferring archived files from smithi037:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721/remote/smithi037 2024-03-29T11:47:57.125 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:47:57.161 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721/remote/smithi146 2024-03-29T11:47:57.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:47:57.196 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:47:57.196 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:47:57.198 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:47:57.245 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:47:57.257 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:47:57.257 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:47:57.266 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:47:57.266 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:47:57.269 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:47:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:47 /home/ubuntu/cephtest 2024-03-29T11:47:57.300 INFO:teuthology.orchestra.run.smithi146.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:47 /home/ubuntu/cephtest 2024-03-29T11:47:57.301 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:47:57.311 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:47:57.330 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:47:57.357 DEBUG:teuthology.nuke:shortname: smithi037 2024-03-29T11:47:57.357 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:47:57.384 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:41.160509', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLoffezH04GjNximqgvmhGPgsmhf2UKf/l7edPWBk4Yd7DA6Ai5XwXMG3gQT7qxKm1AO8CW3cwfU6suMj4+pL/Y='} 2024-03-29T11:47:57.387 DEBUG:teuthology.nuke:shortname: smithi146 2024-03-29T11:47:57.387 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:47:57.406 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622721', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:41.161909', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOgITzAaFXRUJPNt5KLN+jaof4PCZTceu09kE0/gT7yB+Qt2bdVMpg+Ey3GHI4/SO7sl5/Lw3mygW0ryf7Gz3BY='} 2024-03-29T11:47:57.437 INFO:teuthology.orchestra.console.smithi037:Power off 2024-03-29T11:47:57.437 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:47:57.461 DEBUG:teuthology.orchestra.console.smithi037:power off output: Chassis Power Control: Down/Off 2024-03-29T11:47:57.461 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:57.478 INFO:teuthology.orchestra.console.smithi146:Power off 2024-03-29T11:47:57.479 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:47:57.489 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2024-03-29T11:47:57.502 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2024-03-29T11:47:57.502 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:57.528 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-03-29T11:48:01.490 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:01.530 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:01.600 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2024-03-29T11:48:01.641 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-03-29T11:48:05.602 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:05.641 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:05.712 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-03-29T11:48:05.752 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2024-03-29T11:48:05.814 INFO:teuthology.orchestra.console.smithi037:Power off completed 2024-03-29T11:48:05.853 INFO:teuthology.orchestra.console.smithi146:Power off completed 2024-03-29T11:48:05.958 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1279.8098540306091 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=98164ac551f14d6386e6b1b3fc2f6a20 status: fail success: false 2024-03-29T11:48:05.958 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:48:06.032 INFO:teuthology.run:FAIL